Sunday, December 28, 2008

Linux / Garmin GPS60 / JOSM / Openstreetmap quick howto

Here's a quick very specific workflow for anyone wanting to get data out of a garmin GPS60 and into JOSM for editing, it's actually pretty trivial, but I had to dig around a bit. The commands will probably work for other USB garmin units...

1) Look at openstreetmap and find a spot on the map where you know there are missing features



Here's loxley valley, I know that there's a footpath going from the upper half of Myers Grove Lane, down past the Robin Hood Pub, over the river loxley and up the other side of the valley, and it's not on openstreetmap.com. Sweet!

2) Turn on your GPS and go walking, taking waypoint notes along the way :)

The best bit ;)



3) Go home and boot up your fave distro and install gpsbabel (apt-get install gpsbabel for me)

4) Download your trace data (Where you walked, I had to do this as root for dev permissions)

root@bhor:~# gpsbabel -t -i garmin -f usb: -o gpx -F my_track.gpx

5) Download your waypoint data (Your, um, waypoints, I had to do this as root for dev permissions)

root@bhor:~# gpsbabel -i garmin -f usb: -o gpx -F my_waypoints.gpx

6) Load into JOSM and edit / save / upload



When you're done, you can see your trace and waypoint data.



7) Click "File" -> "Download from OSM" to pull down all the openstreetmap data for the area you've walked over



8) Add features corresponding to your trace and waypoints. In my case, a new footpath, a couple of new postboxes, a bridge, 2 schools and a community ctr (and a slight fix to the path of the river loxley so it actually flows under what I know to be a bridge. (I hope thats the right thing to do :S)). Upload your changes to openstreetmap.com

9) Enjoy your work on openstreetmap.com

- Will add the photo just as soon as the OSM render is complete :)

10) Don't forget to upload your postcode coordinates to freethepostcode.org!

Monday, November 17, 2008

Quotes

Read this today and it resonated with me for some reason, worth a read.

“It is not the critic who counts; not the man who points out how the strong man stumbles, or where the doer of deeds could have done them better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood, who strives valiantly; who errs and comes short again and again; because there is not effort without error and shortcomings; but who does actually strive to do the deed; who knows the great enthusiasm, the great devotion, who spends himself in a worthy cause, who at the best knows in the end the triumph of high achievement and who at the worst, if he fails, at least he fails while daring greatly. So that his place shall never be with those cold and timid souls who know neither victory nor defeat.”
Theodore Roosevelt

Wednesday, November 05, 2008

Christmas parties, martial arts and "Stuff"

As it's my personal blog.....

Every once in a while someone brings this old chestnut up, and I thought it was probably about time I aired my recollection....

A few years ago we were out on a pretty serious "Northern" christmas party which had kicked off at lunchtime and gone on well into the evening. The attendees were from a wide variety of backgrounds: our own company, a company we were working with lots at the time, and associated friends and customers of both groups. As the day wore on, one of the party became increasingly agitated. This was, as is always the case, a perfectly nice person who was carrying a bit too much personal baggage at the time and had too much to drink. I can say this, because I can become a morose so-and-so after a few beers, and alcohol affects different people in different ways. Alas, in this case, the beer manifested itself in outward aggression. Initially, this exposed itself as simply trying to pick a fight with two separate people. When it became apparent that nobody was going to give into that kind of thing the mood changed somewhat and we'd gone from slightly charged to grabbing someone (Not me I hasten to add, I only got involved at this point) by the throat with one hand and grabbing a bottle with the other. What followed can only be described as a brief scuffle which ended with my holding onto this person and exchanging a few words. After a few moments I let go and everything was fine. Apologies were exchanged and all was well with the world. No fists were involved, no mention of operating systems (Was made all day long actually), and no real drama.

Some people I know have a huge lack of understanding of the martial arts, and I wanted to take this opportunity to say how proud I am, of the people I train with, in the past and now, and in fact of who I am, and my actions generally. I might be romanticizing a little here.... Firstly, when it came to it, the guy probably wouldn't have glassed anyone. Just after picking up the bottle he might have thought, WTF am I doing, and then let it go. I have to say I think there was a substantial amount of red mist by that point and I don't know what would have happened if he'd been left to get along with it. The way I see it tho, two people were protected that night, especially the aggressor. Besides the obvious criminal implications of what might have transpired, I can't imagine the employers involved could have allowed anyone with a violent crime behind them to continue working in an educational setting.

Whilst I'm on a roll.... Here's a story seldom told before

Many years before, in between my hapkido and taekwondo days I was walking through town, and heading towards the then underpass under arundel gate in the town center. As I walked towards the steps a couple in front of me got to the top of the steps and turned away, opting instead for crossing the road. Hmm... So I headed towards the steps and rounding the corner found four young males standing over a motionless body. It did, in fact, look like a pretty serious mugging. Nowthen, the reason I got into MA in the first place is that my mouth sometimes runs on its own and gets me into potentially silly situations. So I asked what was going on (Well, sorta). It later was explained to me that this is what had really happened........

These five chaps had been to a funeral and gone out to send their friend off in style. In the late afternoon they stopped for burgers on the moor and headed to the next pub. At the top of the steps, one of the five had taken a tumble, bashed his head and was in fact choking quite properly on his half swallowed McDonalds (I was already a veggie by thiis point, but it only re-enforces my conviction ;)). When I got to the bottom of the steps the chap was a beautiful shade of blue, and his mates were in a frenzy. A couple of pats on the back and a little encouragement got him going again, and although still out of it, alive and well at least.

Ok, it's nothing that anyone else wouldn't have done. However, firstly, I'd obtained my first aid qualifications as a direct result of my MA training. At some point, anyone who's serious about training realises that bad things can happen, either in training, or in real life, and if you're serious about protecting yourself and others, you better know what you're doing when it comes to helping people. Secondly, although I would have shouted from the top of those stairs anyway, I was able to do it with some confidence and authority.

So I get a bit disappointed when people make fun of these situations. To my count, my MA training has probably directly saved 1 life, and possibly stopped two serious injuries and two people from trouble with the authorities, as well as minor "Conversations" where someone was sent on their way happy. I've simply walked away from just as many when possible. I've never yet hurt anyone, and I've stood up to people who actually wanted to do me harm, and allowed them to continue along with a different perspective. I don't know what other people would do in the same situations, but I'm pretty happy being self-reliant and knowing I can depend on myself to do the good stuff, and the bad stuff as needed. I'm proud of these achievements, even if people trivialise and caricature them just for effect.

As they say... Talk if you will, walk away if you can, run if you must, but if all else fails; defend yourself


Bah.. End rant.

Tuesday, June 10, 2008

Lucky Guy!

Certainly in the category of personal posts.... :)

I had a bit of an epiphany yesterday, if you can call it that. Sat in the board room of one of our partner organisations with 7 others preparing for a project interview. Suddenly for a second I found myself having one of those reflective out-of-body moments where I stood back and listened to what was going on....

Exactly when did I become someone contributing to senior management meetings in a meaningful way? Here we were, chatting away and discussing strategy. Damn it, it's just not right. I used to have a proper reputation as a pain in the ass developer, where did that guy go? In truth he's still here, but what I realised is that many of the other people in the room I've known for a long long time, all my working life in fact in some cases. And the funniest thing was as each of those people spoke I found myself having the most profound feeling of "I can learn something here". And mostly that was from people who used to be as much of a pain in the ass as I was (am).

All this made me think it's pretty amazing to have stayed so close to one industry and maintain such close relationships with people throughout ones entire working life (to date anyway). I can only hope that my contributions to the group is equal to that of my peers. I'm sure part of the reason for this is Sheffield is closer to being a village than a city. All the same tho, to have maintained friendships and working relationships through the ups and downs of growing our professional lives, and the many and assorted businesses we work in, really is quite amazing and, to me at least, a fine example of what can be achieved with a bit of perspective, persistence and willingness to work at things.

Tuesday, April 08, 2008

Dirty Dirty Hacks : Cheapo SQL Tuning for Hibernate Generated Queries

Today I've had a nightmare trying to figure out why a particular hibernate app was performing like a dog. After an hour of just looking at the logs and finding all the obvious points where the app was struggling it occurred to me that I needed something more exhaustive. Fortunately, having insisted on a high level of coverage for the use cases in the unit tests I'm confident that our unit tests exercise the SQL fully. Hmm interesting. What follows is representative of the pure joy of the unix operating system (Without the dodgy 70's black and white photos!).

So, turn on hibernate SQL logging, you get a mix of logging and SQL, so (We use maven as configuration managemnt)

mvn clean package | grep "Hibernate: select"

Just pulls out the select SQL. Cool, 8.5k bits of SQL from the unit tests, thats a lot.

mvn clean package | grep "Hibernate: select" | sort -u

ooh sort and only output unique lines, down to ~350 statements

Right, the statements have bind variables in them and I want to pipe the statements into MySQL's explain feature, time for a quick script:

#!
# For each statement
while read line; do
# Chop off the start of the line, replace all the ?'s with "" (Because explain plan doesn't care)
sql=`echo $line | sed 's/^Hibernate: //' | sed 's/\?/""/g'`
# For my sanity
echo evaluating SQL: $sql
mysql -u USER -pPASS DB < explain $sql
!!!
echo
done


Cool, take each statement and run it through MySQL Explain Plan. Final phase


mvn clean package | grep "Hibernate: select" | sort -u | perf_script.sh > analysys.txt


Leaves me with a file like

evaluating SQL: select authorityh0_.ID as ID162_, authorityh0_.DESCRIPTION as DESCRIPT2_162_, authorityh0_.TITLE as TITLE162_, authorityh0_.URL as URL162_ from IN_AUTHORITY authorityh0_ where authorityh0_.TITLE="0"
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE authorityh0_ ALL NULL NULL NULL NULL 2 Using where

OK, it doesn't automate the hard part, looking at the plan, but at least it makes the job a damn sight easier. Well, it does for me at least.

*peace* e.

Tuesday, March 18, 2008

Fun with gnuplot



*Disclaimer : I'm a computer geek, not a scientist, the science is probably wrong, but I hope it's clear enough for you to draw a graph that means something*

Like the geek other half of first year biomedical students everywhere (Yeah there must be loads of em) I find i spend far too much time trying to get excel to do best fit lines on data series. And then, after all that hard work, and extracting the straight line formula, plugging in the unknown and getting the value out, what does your beloved do? Use the word drawing toolbox to hand paint a line onto the graph because excel can't do the 1 thing you really want which is a graphical representation of how you arrived at a particular value.

I'm no scientist (By a long way, I daren't even experiment on my own head) but I do know that generally, for solutions of unknown strength you can take a series of samples of known strength, run those samples through some analysis process such as spectrophotometry and then calculate your unknown using the same process. I'm also not the worlds biggest fan of office, and felt slightly let down by OpenOffice capabilities in this area too. Of course I needn't.. like all open source tasks, it's a question of right tool for the right job. instead of spending 3 weeks trying to make excel do something it can't I should have invested an hour in gnuplot. Here's the output of my lunchtime play.

I've got a file called exp_1.dat of space separated values:

1 1
2 2
3 5
4 6
5 30
6 29
7 28
8 27
11 3
22 23
25 27
26 40
27 30

The values are deliberately iffy so I can see the effects of some different operations on the graph. Specifically, the trend of values from 5 to 8 are a descending line opposing the general upward trend of the series. This is to test those occasions where you want to omit the acceleration and deceleration phase.

And here's my script.. it runs as is under linux when gnuplot is installed, i understand there is a windows runtime environment for gnuplot too.

---- Start ----
#!/usr/bin/gnuplot
# My first attempt at a plot that just fits an straight line to the entire set of points
# in the source data file. Best fit, but with just a specific sub-range of data, testing
# best fitting when excluding accelleration and decelleration phases of a reaction.

reset

# Manually force x and y range, can omit, just for clarity here
set origin 0,0
set xrange [0:40]
set yrange [0:40]

# plot of size 800x600
set terminal svg size 800,600

set output 'graph2.svg'

# Our first best fit line, for all data values
b = 0.0
m = 0.0
f(x) = m*x + b
fit f(x) 'exp_1.dat' via b, m

# Fitting only x values between 5 and 8, in practise we look at the values and decide
# where the acceleration and deceleration phases start and end
b2 = 0.0
m2 = 0.0
f2(x) = m2*x + b2
fit [5:8] f2(x) 'exp_1.dat' via b2, m2

# Here is where we'd do a bit of math to re-write y=mx+b the values for b2 and m2
# and plug in our required y value in order to arrive at
# Our test input value is 17
iy = 17
ix = ( iy - b2 ) / m2

# Some output
top_title = sprintf("Test best fit for subset outside acceleration and deceleration");
set title top_title
set timestamp "Last updated: %d/%m/%Y, %H:%M" top

t = sprintf("%1.3fx+%1.3f",m,b)
set label t at 20,16

t2 = sprintf("%1.3fx+%1.3f",m2,b2)
set label t2 at 5,13

set xlabel "My X Label"
set ylabel "My Y Label"

set key left
set grid

# Draw some nice lines from our unknown sample to the calculated absorbtion
set arrow 1 from -1,iy to ix,iy nohead
set arrow 2 from ix,iy to ix,0
# Add A pretty label
a = sprintf("Absorbtion at %f = %f",iy,ix)
set label a at ix+3, 3

# Draw it!
plot 'exp_1.dat' title "Source Data", f(x) title "Best Fit 1" , f2(x) title "Best Fit 2"

---- End ----

The net result is to the right. The script as is outputs SVG, scalable vector graphics, but for display on the blog I had to output as a gif. The SVG is (To me) much nicer looking and it has the advantage of being, well, scalable. Even better, the script is relatively generic, so any data series can be plugged in and all the user needs to do is set the unknown sample measurement in y1 and it will auto draw a line and point out the required x value. Really quite neat, and best of all, can be incorporated into an automated process.

Well I don't know if it's useful, but it beats real work.

Wednesday, February 20, 2008

More crap from facebook

Ok, firstly, facebook.. yes, the apps are poorly behaved, but, and I'm sorry to say this, the architecture and the implementation environment really do encourage this sort of thing, and half the problems with facebook aren't the apps, it's PHP. Yes it's lovely and easy to use, but it's incredibly hard to secure.

Having said that...

Others see you as sensible, cautious, careful and practical. They see you as clever, gifted, or talented, but modest. Not a person who makes friends too quickly or easily, but someone who's extremely loyal to friends you do make and who expect the same loyalty in return. Those who really get to know you realize it takes a lot to shake your trust in your friends, but equally that it takes you a long time to get over it if that trust is ever broken.

More nonsense that I hope turned out to be more or less on the button :)