Pruning Large Files?

[loo_hoo_ser]

Hi,

I've started logging temperature data by opening a text file and appending the new temperature reading to it. Eventually it will grow very large, if left unchecked.

What I want is a way (using VB or some external program) which will allow me to truncate the file after a period of time has elapsed (say 2 weeks). I don't want to simply delete the file and start over - a graph is created based on this file and I'd like the graph to appear continuous, with the old data being thrown out (disappears) after a period of time.

Ideas?

[Osler]

A little more info would be helpful. A simple standalone VB program can be easily created that allows for onscreen graphical display of the temperature (acquired from HB at a set time interval) with this program also opening a text file to save the datapoints as they come in. The data file could be named for the particular day of acquisition, that way a days worth of data could be reviewed at a time . I have a similar program that could be easily modified to do this....its just a matter of finding out the details of what you want and how you want it displayed (i.e., the program can draw a continuously updated line graph while simultaneously drawing a thermometer for the current temperature reading).

Osler

[loo_hoo_ser]

A simple standalone VB program can be easily created that allows for onscreen graphical display of the temperature (acquired from HB at a set time interval) with this program also opening a text file to save the datapoints as they come in.

I already have a program to plot the data from a input text file - I am using gnuplot to plot the points by using the execute program device in HB. Like you suggested, I have a task that wakes up and writes the temperature to a file. More specifically, every 7 minutes, the task wakes up, takes a reading from the X10 temperature sensor and appends to a file the temperature, the current high temp, current low temp, and of course, the date and time of the reading (all as ASCII text).

I used a simple file open with append in my VB script and it adds the new data at the end of the file.

I'm not proficient enough with VB to know how to manipulate files (e.g. seek operations). I was thinking I would create a separate task that would wake up once a week and prune the file. But what I really like to do is after about 700 readings (a week's worth of temperature data roughly) is that I'd like to push out the old reading (e.g. get rid of reading #1) and the new reading becomes #700 (it would have been #701 but since #1 is gone, it's now #700). This way, I have an effect of a moving plot that shifts slowly to the left as time moves on.

However, I fear this might be too cumbersome unless there is a way to do it in VB by using file seek operations to quickly manipulate file structures without having to read through the whole file.

Thanks for any suggestions you can provide.

P.S. how would you have generated a plot in VB that Housebot could display without using gnuplot?

[Osler]

Would you be averse to using a database for storage as opposed to a text file?

The graphing code I have currently only graphs in the stand-alone .exe. In its current form it could not be displayed on a software remote. However, I think there may be a way to do it similar to what you describe in gnuplot.

Osler

[Osler]

Try this and see how it works. The graph will display 100 datapoints at a time, marching to the left. Each datapoint is stored in a database file, with the maximum number of records per database file capped at 700. The program will generate a new JPG graph with the addition of each datapoint. This graph can be displayed in a software remote. Example graph below (colors, etc. are customizable):


(dashed lines represent target temperature and range)

The program and needed files can be downloaded from here:

http://home.comcast.net/~jpbrowning2/TempChart.zip

This is very rough but gives you an idea of what can be done.

Osler