HOME  |  NEWS  |  BLOGS  |  MESSAGES  |  FEATURES  |  VIDEOS  |  WEBINARS  |  INDUSTRIES  |  FOCUS ON FUNDAMENTALS
  |  REGISTER  |  LOGIN  |  HELP
Comments
You must login to participate in this chat. Please login.

hate the idea of tweak and recompile (tho understand the desire to keep it simple), like the idea of engine (any) simulator

 Good techniques!  --  Would like to see:

     * Design Reviews (multi-disciplined, at various phases)

     * Code Reviews / Structured Walk-Thru's

     * Peer Reviews

Maybe you include this is in the design phase, not "test" phase ...

 

 

Good presentation! typical flow for development and testing at high tech industry

In my experience some dev-engineers do rely completely on test-engineers to find bugs and report them back to dev-engineers.

I have enjoyed well prepared thought thru slides.

Iron

Very good tips.  Enjoyed the session.

Iron

This hackware idea sounds alot like test driven development

 

I see these things alot while working

with systems and know they have a tendancy to

pop out of the woodwork when you actively look for them

I agree very highly on the overnight stress testing

to find intermintant errors

Yesterdays was great

Looking Forward to today

oops wrong presentation.

Iron

thanks, good presentation

Iron

@clia (and @THasham): There are several books out there dealing with various aspects of software/system testing. I have not been out looking for what's there so I can't say. I would guess that there is not a book out there about Hackware Testing or similar. It may be because it violates some good practices in software development.

Iron

@jrjohns and @caa028: I didn't use the term, "hackware," back then. In fact I didn't realize that what I was doing was that unusual. So in trying to describe it to others, "hackware" was what I came up with because I hacked up the code real quick to run a quick test, then I took it out. So it never became a formal development/test methodology deployed within HP. But yes, "hackware" does have a rogue or underground connotation. I could not find online the term "debug private" used in this context. Another name that describes it is "ad hoc" testing but that is already used and has other meanings within the testing field. Here's some other possibilities: "precision testing," "fleeting testing," "temporary testing," "short-term testing," and "in-code testing." None of those have the same impact that "hackware" does. Any ideas out there?

Iron

@caa28: Yes, you can induce some errors at the system level. We could cause a paper jam to occur in the printer but we could not cause the fuser to overheat or the laser to stop working without extensive modification to a pristine system.

Iron

@CNorton; NSWC PHD, CSSQTs for CG-47 B/L 3A and DDG-51 class Flt II/IIA.  CNO Project 801 Systems Interoperability.

 

Iron

thank you great presentation

Iron

make it work first, then make it better. sounds like Gary's comment about getting it to work the first time.

Iron

@Gary, sorry, haven't used hackware to your extent... looking forward to the next time though.

Iron

@Gary, I like the hackware technique. I've used variations of it, but to your extent. Like any tool, it has a place when and not to use it.

Iron

Great stuff! Thanks Gary.

Iron

slide 10, stress testing begins with the belief that the team really wants to find a problem, any problem before the product is released.

Iron

looks like a lot of chatter about hackware.

Iron

@GStringham - Got it.

Iron

@Ran: I also didn't get any awards for having talked to the hardware engineers 18 months earlier which allowed me to get my driver out on time because I didn't have to mess with working around hardware defects.

Iron

Hi Guys, better late than never.

Iron

@GStringham - You bet, bro.  You earned it.

Iron

@Ran: Wow! Thanks for the high compliments.

Iron

@THasham: I have not researched books on the subject. Sorry.

Iron

@snandu13: Humidity is a good way to stress physical items, like the hardware and mechanical components. Most of what I was talking about was stressing the software/firmware aspects. It is rare but I have dealt with cases when humidity affected the firmware, such as changing the timing of mechanical or electrical parts.

Iron

@JeromeG12: In most of my career, I used C for the production code language. While everybody thinks that C+ is going to take over, it hasn't yet in the embedded world.

Iron

@luizcosta: No, I do not know how Apple does testing.

Iron

@luiscosta: "Fuzzing," a term I had not heard before. There is a Wikipedia entry for it. http://en.wikipedia.org/wiki/Fuzz_testing. But it looks like it is stuff external to the unit being tested by hammering it from the outside with a bunch of random parameters. That, to me, is very similar to the Stress Testing I was talking about. My hackware is internal to the module being tested and it does a specific task (not random.)

It doesn't make sense to version control hackware. For example, I may, on the fourth loop, pretend that the hw returned 0x3D. I verify it works, then I change it to 0x7D, recompile, and run it again. I may run through a dozen values in 30 minutes to complete my test. Submitting each one of those to version control doesn't seem practical to me. And, if I were to make that a unit test, I'd have to deal with a list of desired values and a loop of some sort. Again, my hackware tests the finer details of my code, details that are so minor that most people probably wouldn't make a unit test for it.

Iron

Kinda looks like Gary "has left the building."

Iron

@Mr. E - Yep.  It's all good.  Whatever is most appropriate will work fine.

Iron

Expanding on the #define DEBUG <level> idea: one can have a global debug variable which is a bitmask, each bit of which controls a particular class of debug information to allow selection at run-time. This global variable can then be set via an initialization file and/or adjusted via a console command, depending on what is available in the system.

Iron

Thank you again Gary; some very interesting thoughts and techniques here.  I wanted to reiterate THasham's earlier request for book recommendations.  I have to run now, but I can check back later here or during tomorrows lecture for your response.

Iron

So long Alex, Gary, and everyone else.

Iron

OK folks, looks like this is it, and I need to take off.

Iron

Good points on using #define DEBUG <level> to control how much debug code is in there. I also used a global variable that was accessible from the debugger to control the level. By default, it would be low debug output. But if I wanted to, I could break into the debugger, change that global variable, then have more verbose debug output when running a test. A global variable also allowed hackware to temporarily turn on a higher verbose level when desired.

Iron

@luizcosta: Regression testing where there is a list of tests that get run through every time. It is used, for example, every week to qualify a new code drop before releasing it to everybody, to make sure nothing is broken. It is a subset of all the tests because it is mainly a quick test to make sure nothing is generally broken.

Iron

Thanks Gary . Very interesting  lecture and I had really good exposure since I am only a hardware engineer all through my career

Iron

@syakovac: You brought up the ATPG scan chain to force a state. And it made me realize that the primary focus of today's lecture was on software. I neglected the hardware side of the equation. Sorry about that. But yes, when testing hardware, my hackware principles still apply. Come up with a quick and dirty scan pattern, run it through and test. What I was referring to with states was software states, where it is easy to write to the state variable to put it in a desired state.

Iron

@GStringham – When testing code during the writing of code (hackware), I found it very useful to build into my code compiler directives (at the top of the code) for conditional compiling.  That way, I could leave the test code in the final version if I wanted, or I could take it out if I wanted.  It really didn't matter because I could set the compiler directive to TRUE or FALSE.

Iron

@s.schmiedl: You asked why hackware instead of doing it right the first time by writing a unit test. Good question that seems contrary to what I said. I can test things in finer detail with hackware than I can with unit testing. If I had the time to create a unit-level test for every type of hackware test case I run through, I would create so many unit-level tests that there would not be enough time to run through them all more than once or twice before the product needs to be shipped.

Iron

Thank you. Gary and Alex.

Iron

Yes. It is true that DOE will be greater use for improving test effectiveness. But it requires a good amount of statistical knowledge and good training.

Iron

@GStringham - You are so correct when you say there are no awards for producing code with zero defects.  On more that on occasion I have gone to unit integration with the other programmers on the team and been the only contributor with zero defects, but nobody recognizes the effort/results.  No, there are no awards!

Iron

I wanted to find out how much attentions validation time should be planned for DOE's (Design Of Experiment).

Testplaning - there are planning factors that need attention on repeatability over multi-runs, VT's and other varriables. Those thoughts will add or reduce validation time, resorces, and other infrustructure. I rely on DOE's to build statistics and modles to help focus on worst case areas. This gives many advantages like varriablility, repeatability, sensativity to help focus on more or focus less on particular areas. If you can not test milions of chips, platforms, and devices you need to plan a statistical target on exceptable margins. You can test ever thing so how do you plan and take some risks.

Iron

@s.schmiedl: I don't agree that ALL test code has to be extracted from the product before shipping to customers. But it definitely must not cause a hinderance to the performance of the product. You can tell that Microsoft leaves test and debug code in because when some app crashes, it asks if it can send the data back to MS for analysis. This really helps them understand what is going on out there.

Iron

You confirm that most of what we do here is on par.

Iron

READING: http://en.wikipedia.org/wiki/New_product_development

Iron

@GStringham - I can't speak for anyone else here in our forum, but I found today's session with you the best session for me since we started back in January.  Thanks Gary for such a good session.

Iron

@clia: Yes, some companys do throw out a little tested product and let customers test it. It's not a business model I care for but it does work for some.

Iron

Hi Gary..Do you have any recommandation for books on the subjects? Thanks.

Iron

In stress testing did you have to do the system testing at eevted temp. and different levels of Humidity?

Iron

Hi Gary, I would like to know what is a general production code language to gain familiarity with?

Iron

GARY: Do you know what is Apple doing on "testing"? 

 

Iron

Gary - Thanks for an interesting lecture.

Iron

Thanks Gary for an interesting lecture

Iron

Thank you Gary and Alex.

Iron

Thank you Gary, very good tips for testing and debugging.

Iron

GARY: If you use "HACKWARE" isn't better to incorporate that in the versioning process where the "librarian" of documents keeps in track of all inserted "bugs" used during the test phases? Another point is "fuzzing" is already used instead of "hackwaring" and there are books and research done on it..

Iron

Thank you very much for today's lecture

Iron

Would it not be better if we use start-up diagnostics and other built -in diagnostics of all the critical functions ? Unfortunately the diagnostics is never complete in a system

Iron

@Andoglehog - WTH was that passage?

Iron

Testplaning - there planning factors that need attention on repeatability over multi-runs, VT's and other varriables. Those thought add or reduce validation time, resorces, and other infrustructure. I rely on DOE's to build statistics to help focus on worst case areas. This gives many advantages like varriablility, repeatability, sensativity to help focus on more or focus less on particular areas.

Iron

@jrjohns - call it "debug private" (as private build)... very common name ;)

Iron

Thanks for another great session Gary!

Iron

It's too bad that the name Hackware was chosen.  My management chain would have a fit if I suggested taking a Hackware approach (without knowing what Hackware was.)  Perhaps a more subtle name?  :-)

Iron

Thank you Gary for the excellent methodology for testing systems. 

Thanks gain for a great session, Gary & Alex... thanks caa028

Iron

Great Session Gary!

 

Iron

You say you took the hackware out - why not control the test code using compiler directives or #defines and utilize a "test" branch in your code repository?

 

Iron

Thanks for the lecture, Gary.

Iron

Thanks for a great session!

Great Lecture..Thanks much again..Gary and Alex.

Iron

Some very good ideas, Gary. Thank you

 

Iron

 a common practice - use various debug levels in production code (controlling what piece of "hackware" is active at runtime)...

Iron

has anyone else been using hackware for testing?

 

Iron

@luizcosta its something that you tested before, but in a new release you test to confirm its Ok

Iron

For manufacturing testing of digital logic, the part is configured as a shift register so that all registers are accessible, controllable, and observable in all states

Iron

@luizcosta - re-running tests that have passed before after touching any other piece of code

Iron

@syakovac..what is ATPG scan??

Iron

What is regression testing?

 

Iron

@DanSchwartz..AGREE!! Been through..

Iron

Just use the ATPG scan chain to force a state if it is that important.

Iron

@s.schmiedl - again, ciq had production code (not hackware) data collection agent... in telecom it violates user consent requirement...

Iron

how do you test the code later again, if you take out the hackware? why not do it right and write a (unit) test for every code path as you go along and create them?

Iron

caa028: yes, but as it's so useful data for "knowing what's going on", the temptation to leave it in is high, I heard Gary salivating about it.

Iron

Don't forget to remove the Hackware when done!

@jmh- With the option to stream out data from a serial port, it sounds like your company also thinks the product should be tested before shipped, unless you do that just for the customer.

Iron

@s.schmiedl - ciq was not "testing" the smartphones, they were collecting data for the network optimization...

Iron

@ jmh - you don't want to have just your customer feedback.   they are screaning when they have a problem, not when everything is ok.  you need to do your 'homework" first in house. 

 

Iron

@jmh - customers supposed to receive a tested product (after the system level testing is complete)

Iron

Re carrier iq: how do you _make sure_ that no active test code arrives in production systems

Iron

That line i don't think even exsists...lol

Iron

@jmh- I agree; they are a tremendous resource! But, my point is that they should not be the only resource.  I cannot see this as anything other than short sighted.

Iron

@Greekeng- But, where is that line drawn between risking losing clientele for missing the almighty deadline, vs. risking losing clientele for putting out sub-par products?

Iron

Yes then after collecting all this data, I had to "de-engineer" the data so management could understand, then they'd cut my time, pulling my hair out.....

Iron

@clia - Best feedback comes from customers!

Iron

We have option to stream data out a serial port in our products for testing and performance purposes.  We then organize it, plot it, save it using excel.

Iron

@DanSchwartz- That is just disturbing!

Iron

I had to create PCbs for communications, servo drive, communications, for a product within 3 months...very little testing,,,

Iron

There is usually a push to get a product out before all testing can take place.

Iron

http://www.advancedbionics.com/

You should see how little stress testing is done with implanted medical devices, such as cochlear implants. Just ask Advanced Bionics!

I have heard that some major companies do very little testing with their products before the first release, leaving us to test for them

Iron

inducing system level errors is ok in system level testing (e.g. system level overload)

Iron

Good Day to you all. It is 70 and sunny in VA.

 

Iron

Good afternoon Gary.

Iron

It's summer in Chicago (for the past week)...

Iron

Hi, Alex.  Good afternoon.

Iron

It is 80 and sunny in Chicago

 

Iron

yes you are funny, but for a Canadian living down here.. it's great, lol

Iron

Sorry for th font, the system did it, not me

Iron

It's snowing in Boise.

Iron

gpapich,

I spent a few years in the Navy, testing at PMTC and NAEC.  Where were you working?

Iron

@Greekeng - Hey there snowy, cold Tucson from wonderfully warm 70 degrees Maryland.

Iron

Plan A, B,  how many more?

@GGtringham, Thank you. 

Iron

@MazianLab: I know there are some in May. They just don't have it up there yet.

Iron

Good day from Tucson

Iron

Alex, acording your Curriculum Calender last session is on April 27 / 2012. Is there any remaining session in June?

Iron

Good afternoon all...

Iron

Good afternoon, folks!

Iron

The next session will start in about 35 min. The timestamps on this chat are EDT (at least on my browser sitting here in MDT.)

Iron

What time is it now in terms of EDT?

Iron

Looking fwd to this.  Did test plans for the Navy as a T&E Project Officer.

 

Iron

Please join our Digi-Key Continuing Education Center LinkedIn Group at http://linkd.in/yoNGeY

Blogger

Be sure to click 'Today's Slide Deck' under Special Educational Materials above right to download the PowerPoint for today's session.

Blogger

The streaming audio player will appear on this web page when the show starts at 2pm eastern today. Note however that some companies block live audio streams. If when the show starts you don't hear any audio, try refreshing your browser.

Blogger


Partner Zone
Latest Analysis
Adam Berger hacked a computer keyboard into a mini key-tar to play with his band.
Altair has released an update of its HyperWorks computer-aided engineering simulation suite that includes new features focusing on four key areas of product design: performance optimization, lightweight design, lead-time reduction, and new technologies.
At IMTS last week, Stratasys introduced two new multi-materials PolyJet 3D printers, plus a new UV-resistant material for its FDM production 3D printers. They can be used in making jigs and fixtures, as well as prototypes and small runs of production parts.
In a line of ultra-futuristic projects, DARPA is developing a brain microchip that will help heal the bodies and minds of soldiers. A final product is far off, but preliminary chips are already being tested.
If you're planning to develop a product that uses a microcontroller, you'll want to take note of next week's Design News Continuing Education course, "MCU Software Development A Step-by-Step Guide."
More:Blogs|News
Design News Webinar Series
9/10/2014 11:00 a.m. California / 2:00 p.m. New York
7/23/2014 11:00 a.m. California / 2:00 p.m. New York
7/17/2014 11:00 a.m. California / 2:00 p.m. New York
6/25/2014 11:00 a.m. California / 2:00 p.m. New York
Quick Poll
The Continuing Education Center offers engineers an entirely new way to get the education they need to formulate next-generation solutions.
Sep 22 - 26, MCU Software Development A Step-by-Step Guide (Using a Real Eval Board)
SEMESTERS: 1  |  2  |  3  |  4  |  5  |  6


Focus on Fundamentals consists of 45-minute on-line classes that cover a host of technologies. You learn without leaving the comfort of your desk. All classes are taught by subject-matter experts and all are archived. So if you can't attend live, attend at your convenience.
Next Class: September 30 - October 2
Sponsored by Altera
Learn More   |   Login   |   Archived Classes
Twitter Feed
Design News Twitter Feed
Like Us on Facebook

Sponsored Content

Technology Marketplace

Copyright © 2014 UBM Canon, A UBM company, All rights reserved. Privacy Policy | Terms of Service