⛓️💥 Putting It to the Test
What it Takes to Break Things in a Useful Way
🦸 Guest Writer: Daniel
This week’s article is a special edition written by Daniel Lieben about test engineering. Daniel completed his BSME at Northeastern University in 2023, doing two internships in his time there including one at Tesla. That internship led to a full-time role with Tesla as a test engineer, focused on their vehicle heat pump systems.
⚙️ Mechanical Engineering Resources:
We have put a dozen guides for mechanical engineering students and early professionals on our website
50 Hardware Startups who have raised less than $50 million (perfect internship targets)
How to handle The Behavioral Interview
What it takes to be The 10X Intern
Why Should Hardware Teams Run Tests?
This may seem like a silly question with an obvious answer, but it’s far from it. The easy answer is “so we know the product works,” but in practice, “works” is both very vague and a very low bar. Does it work for the entire intended lifespan? Does it work in all environmental conditions? Does it work when the user abuses it? These are all better questions to ask than simply “does it work?” And luckily - tests can answer all of these questions if executed well.
While there are many factors that make a useful test, there are two golden rules that I believe are non-negotiables:
Tests should answer questions. A test is useless if it doesn’t yield new information. Questions should generally be as specific as possible. For example, if you wanted to test the range of an electric vehicle, here are some questions you could ask, from least useful to most useful:
I don’t have a question, but my boss says this test needs to be run (0/10, you don’t even know what you’re looking for)
How far can this car go on one charge? (3/10, at least it’s a question, but it’s very vague)
How far can this car go on one charge on a flat road at 70 mph on a 20°C day? (6/10, good specificity, but how useful is the result?)
What is the car’s range sensitivity to road grade between -5 and 5%, vehicle speed between 40 and 80 mph, and ambient temperature between -20 and 30°C? What’s the best-case and worst-case range within those constraints? (10/10, now that’s a question that provides useful information)
Tests should be actionable. This one is a pet peeve of mine. If your project’s next steps are the same regardless of a test’s outcome, then it’s a useless test. Ideally, before starting a test you should know what all the possible outcomes lead to, and what defines each of those outcomes (i.e. your pass/fail criteria). I’ve watched engineers spend weeks setting up and running tests, only to be told that the project is proceeding as-is despite their subpar result - fantastic way to waste time and money.
How to Run a Test
Now that we know what makes a good test, how do we actually run one? Testing is extremely broad and not easily standardizable, but almost all tests will have a few components in common:
Your device under test (DUT). This is the part/system that is actually being observed; the thing that you have a question about. In the range test I used as an example before, this would be the car itself.
Instrumentation. Instrumentation has two buckets, sensors and data acquisition (DAQs). If you need to measure forces, temperatures, pressures, or whatever else, you’ll need to pick appropriate sensors for the job. DAQs are devices to read the sensor’s raw output - usually a small voltage or current, but there are more complicated types - and translate it into data that can be read/saved on your computer. Instrumentation is a very deep topic that I don’t have space to do justice in this article, but people make whole careers out of it. In the range test example, this would be an inclinometer to measure road grade, a speedometer to measure vehicle speed, and a thermocouple to measure ambient temperature, plus DAQs to read them all. For shock and vibe do NOT underestimate the importance of sampling frequency.
Computer & test script/software. Basically all tests will involve using a computer to do some of the following: log data, control the DUT & its environment, detect test failures/issues, track test progress, apply safety controls, visualize data, and more. Sometimes there are off-the-shelf softwares that can fit your needs, but often it’s necessary to write your own code to do exactly what your test calls for. Python is fantastic for test scripts, as there are tons of pre-existing libraries you can use to communicate with any given device in your test setup. In the range test example, you’d probably want to write a script to log all your sensor data over time, plus some values like battery percentage from the car.
Integrating these pieces together takes some persistence when you’re doing it for the first time, but gets quite a lot easier each time you go through the process. There is also a balance to be found between automating 100% of a test versus keeping a human involved in it, which brings me to my next point.
Test Automation
Let’s say you want to run a test to answer the question “will my part survive its 20-year lifespan with no degradation to its performance?” The test to answer that question will take quite a long time to run - not 20 years, but at least a few months (fun fact: I have a test that has been going for two years!). It would obviously suck to manually operate a test for that long, so this is a case where fully automating the test makes sense. How does one actually do that?
The most important thing that changes between a manual test and an automated test is safety. An automated test will continue to run even when you’re not sitting there babysitting it, so it needs to be safe for passersby in the lab as well as the lab itself. Automatic failure detection is a must - if anything goes wrong with the test, the test script needs to shut down the DUT safely and dissipate any hazardous energy, whether that energy is electrical, mechanical, thermal, or other.
Aside from safety, things like test tracking and data logging also become critical. If something interesting happens in the test and you’re not there to watch it, you need to be able to dig into the data and see exactly what happened and when.
Late parts don’t have to be a surprise
The worst part about a missed deadline isn’t the delay, it’s finding out too late to do anything about it. Black-box sourcing platforms give you a lead time estimate and then silence. On Jiga, you communicate directly with the manufacturer and get proactive updates on production status. When something shifts, you know early enough to adjust, not after your team is already scrambling.
Why Test and Simulation are Besties
Although test and simulation seem like two unrelated subdomains of engineering, they actually go together very well. Simulation serves a similar purpose to testing: to answer questions. The methodology is different, but the goals are the same. As for how they can work together, the short version is: tests validate simulations, and simulations extend tests.
For example, let’s imagine you want to figure out how hot the inside of a car will get when it’s parked on a sunny day. This is a relatively easy test to run - grab your car, put some thermocouples in it, park it in the sun, and let it sit for a few hours. Simulating it, on the other hand, is pretty complicated. A car is a huge system with lots of thermally coupled parts, and you’d need to model all three modes of heat transfer - yuck. To make an accurate model of that without any test data to compare to would be a huge challenge for even the best simulation engineers. With test data though, it’s a different story. It’s much easier to make a low-fidelity model that you can tune to match the test data you’ve collected, and then you can be confident that your model is accurate.
Next, let’s imagine you want to figure out how hot that same car would get if all the windows had 5% higher reflectivity. This is much more of a pain to test; you’d need to get new custom-made windows and install them on the car. However, if you have a model that you know is accurate from the previous test, you can just change the windows’ material properties and rerun the simulation! In this way, the simulation has extended the test. Note that the further you extrapolate the model from the test data, the less accurate the model is likely to become.
Why Testing is Awesome for Early-Career Engineers
Many people say that test engineering is a very good early-career job for a mechE, and I wholeheartedly agree. Test engineering is inherently very generalist; in my 3 years at Tesla I’ve worked with mechanical hardware, electrical hardware, firmware, reliability, failure analysis, software, simulation, and more. It’s a nice blend of hands-on lab work and computer work, and there are usually travel opportunities for tests which require specialized facilities. Even if your long-term aspirations are in design, test engineering is a great early-career stop to teach you about what makes a design good, though some engineers have reported difficulty switching between disciplines later.
Thanks for reading, shoot me an email at hardwareishard@gmail.com if you have any questions or follow up content about test engineering you want to see!