How to Process, Filter and Clean Yield Monitor Data
January 16th, 2008
Hi Everyone – long time no post, I realise.
Ok – so a bit of an update – I’ve spent the last little while learning a new language (VB.NET). It’s been a nice challenge, as most of my programming to date has been non-object oriented stuff – command line interfaces or totally embedded solutions usually written in ANSI C.
I’ve been up in North Queensland helping a company up there develop yield maps for their Sugar Cane harvesters (actually, I helped design the equipment that sits on the harvesters and does the yield monitoring, as well). The process of taking the raw data off the harvesters and converting it into yield maps is a fairly long and drawn out one. Firstly you need to filter the data (something I had been doing in Excel) to remove erroneous positions and obvious outliers.
Secondly you need to convert the latitudes and longitudes from the yield monitor into eastings and northings (A degree of latitude isn’t the same distance at every position on the earth) so that you can project the data onto the local industry GIS system (the projection used is called the Geodatic Datum of Australia – GDA Zone 55) which was a bit of a mission – I had to convert a long and convuluted mathematical process (about three pages of equations) into a neat little visual basic function.
Thirdly you need to tie the data back to actual weights to obtain a calibration for each machine (had a hybrid Excel / Access database for that job) and fourthly you need to have a way to automagically recalibrate the harvester as the season progresses and the pressures that the yield monitor uses to monitor cane throughput start to change (mostly they go up – wear and tear on the harvester).
I was planning to teach my clients how to do this all manually, but, since I had a hankering to learn a new language and wanted to make life easier for them, I asked a famous Googler what object oriented language he would use. He suggested visual basic, which suited me fine because it turns out that it’s quite similar to C in many ways.
So.. now.. a month later I know a new language and have a great little humungous larger and more comprehensive than I ever imagined software package.
The package condenses what took me about 7 or 8 days to do using excel spreadsheets etc into an algorithm – essentially distilling it down into a 5 minute batch process. The danger is that it will look TOO easy and my clients won’t appreciate the effort that’s gone into it 😀 – but, meh, the satisfaction of a job well done was worth the extra effort.
Once the yield data has been processed by the software, it then connects into a GIS system which converts the point yields into a smooth surface – a yield map. The one I use alot is called ArcGIS, and it makes life rather easy. Specifically, it uses a function called ‘Inverse Distance Weighted (IDW) Interpolation’ to draw the map. Basically, for every recorded point from the datalogger, it looks at 8-12 other nearby points and uses them to ‘guess’ what the yield may be between the points – and hence comes up with a nice smooth yield map.
That process can take a long time – for a years worth of data for 23 harvesters in Australia the process of calculating the IDW maps took me about 3 days with my Opteron 165 based system – which leads me into my next post – about my new computer (feeling very Geeky here).
Entry Filed under: General Discussion
If you found this page useful, consider linking to it.
Simply copy and paste the code below into your web site (Ctrl+C to copy)
It will look like this: How to Process, Filter and Clean Yield Monitor Data
17 Comments Add your own
1. JohnMu | January 20th, 2008 at 6:52 am
Who is this architect giving you crazy ideas?! 😛
Glad to see it’s working out 🙂
2. esmail khorasani | March 12th, 2008 at 12:14 am
Hi dear Cheers
thats very nice
I am a Phd student in agricultural machinery
my thesis is about sugarcane yield mapping
I must develop a similar program for my PC panel that manage the yield monitoring system in harvester
I will be pleased if you send me the source code of this program in VB
this will help me alot
thanks again
lets be in cintact
3. Randy Price | April 13th, 2008 at 12:35 pm
That’s pretty cool. I build and sell a yield monitors too – http://www.bae.ksu.edu/precag/index_files/Page558.htm. How does yours work?
4. theDuck | April 14th, 2008 at 1:12 am
Hi Randy!
Actually, you’ve referenced my yield monitor in your paper. It’s the AgGuide Yield Monitor. I was employed as an independent consultant to conduct trials on various YM arrangements. As such I’m a bit restricted in my ability to fully discuss the mode of operation as some of it is commercial in confidence.
I have been working with cane harvesters for the last ten years, with my specialty being large real world field trials of new innovations. I also developed a full hydraulic model of the interactions between the feed train and chopper in modern billet harvesters, and I can confirm that the graph on http://www.jaisaben.com is a graph of actual field results.
We conducted fully replicated trials using a randomised complete block design differing things like billet length, pour rate, extractor speed, secondary extractor speed etc to ascertain the most repeatable measure under all conditions (the variation in that graph incorporates those various treatments). Each point represents a 4 tonne bin.
Basically that YM uses hydraulic pressure transducers throughout the machine and measures temporal flux in those pressures and between sensors. From this we’ve been able to produce a very highly repeatable and robust YM.
The problem with the optical approach (we tried something similar) js that the bulk density of cane differs enormously between cultivars. The problem with other approaches (roller opening) is that as you double the billet diameter, you quadruple its volume, and, again, bulk density differs dramatically between cultivars and crop conditions.
M
5. theDuck | April 14th, 2008 at 10:47 am
Hi Esmail,
Unfortunately whilst I understand your predicament, I’m not able to pass on the source for my program for a number of reasons that I hope you’ll understand.
1. As a consultant, It’s one of my sources of income at the moment. The program took ~ 3 months to write, and the principles and algorithms it uses longer.
2. I’m also doing a PhD and it will form part of the IP for that PhD.
Cheers and all the best,
Matt
6. Randy Price | April 26th, 2008 at 3:52 pm
That’s a good idea. Nice and easy to put on and not much to break or get dirty. What ever happened to the other Cox idea with the weight plate? Is it still being considered or did it die out for some reason?
Thanks,
Randy
7. theDuck | April 26th, 2008 at 4:17 pm
Randy – Cox’s fell by the wayside as it didn’t follow the KISS principle – also the weigh plates tended to have real difficulties with clogging and abrasion in moist conditions.
Cheers,
M
8. Ryan | April 28th, 2008 at 6:02 am
Hi
I’m very interested in what you are doing here. I have heard of your work before. I’m currently working on harvesting efficiencies myself, just on a bigger scale. My company is currently in the early stages of building a two row harvester. I would like to know more about your experances with cane harvesters. Anyway get in contact with me if you would like to find out more.
Ryan
9. theDuck | May 2nd, 2008 at 12:20 pm
Ryan I’ve sent you an email ~ 1 week ago with my contact details.
Cheers,
M
10. Mukonyole | October 16th, 2008 at 4:53 am
Am glad to have read your document. am a geomatic student doing aproject in sugacane contracting gis for a sugar company. I request u to help me with technical skills like programming of Arc Objects etc
11. Trevor | March 5th, 2009 at 9:50 am
Thats cool mate…I am in north Q too..having issues with yeild mapping using a techagro system…what yield monitors are you using data from to create yeild maps
Cheers Trev
12. Miguel | May 18th, 2009 at 7:58 am
Hi M!
Nice to know that you manage to develop a soft for AG yield map processing! Last output I leaned for you was from our joint Mackenade trial. Does it works specifically with AG yield monitor or can be used or adapted to others e.g. Techagro?
Greetings Miguel
13. theDuck | May 19th, 2009 at 1:11 pm
Hi Trevor – the yield monitor I use and helped develop was an agguide yield monitor. Cheers – M
14. theDuck | May 19th, 2009 at 1:14 pm
Hi Miguel!
Great to hear from you 🙂 Are you still in Ingham?
I deliberately wrote my software in a particularly modular form. It could easily be adapted to process data from almost any yield monitor.
I was planning to do that in my PhD, but due to problems defining the scope of my PhD, I withdrew and have now moved on to other projects 😉
Still available to assist on a consultancy basis with YM workj though.
M
15. Miguel | May 20th, 2009 at 5:44 am
Hi Matt,
Nice to know about. Currently I’m in Cuba working in some projects here and in Brazil.
We also have redesigned our software and have integrated it in a suite called CERES. All programming is being done in Java with open source software.
Now we want to
I’ve trying to access the web link provided by Randy Price and can not access the page.
Could I get in contact with Trevor at NQ?
Keep in touch,
Miguel
16. Lawrence | June 18th, 2009 at 11:40 pm
Hello Miguel, Trevor and Matt. Interesting reading and interesting that you have all ended up on this web page. We conducted series of trials assessing both the AgGuide and Techagro systems together last year under Rob Bramley’s SRDC project. More trials in 2009. The results will be out later in the year. Yield maps for the Herbert will be posted on the HRIC website very soon. The maps will be a collation of Techagro and AgGuide yield maps for the region in 2007 and 2008. Yield monitoring is progressing in the Herbert.
17. Terry Griffin | January 14th, 2011 at 6:15 am
I am interested in your research and wonder if you could share your dissertation or other papers regarding automation (or automagician) of yield monitor data. I work with yield monitor data in the USA.
Leave a Comment
Some HTML allowed:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>
Trackback this post | Subscribe to the comments via RSS Feed