SAMPLING ISSUES - Groups and velocity measurements

[DaveMc]

Hi everyone,

One of the biggest challenges for shooters is to do meaningful testing without burning out barrels and chewing up components. We have become accustomed through need to shoot 3,5 and 10 shot groups to try and determine our optimum loads and same can be said for velocity sampling. There are some major issues with this but when I try and explain this in terms of statistics most peoples eyes roll back in their head and I can hear the snoring before I get the words out of my mouth.

SOOO in an attempt to demonstrate some of these issues I have adapted one of my models to simulate shots on paper. I was surprised myself how well this demonstrates the sampling issues we have when doing load development. exactly the same issues apply to sampling velocity measurements.

i will start with a couple of screen shots (one at a time for discussion) before asking Alan to post the models.
First is a screen shot of 6 targets with 6 * 3 shot groups - I will ask for comments first - Which group / load would you choose and why??

[jasmay]

This within the first post I can see is going to be a valuable thread that many of us will find useful.

Without more info on what's what with your targets, i.e what order they were shot in, conditions, speeds the only thing I currently have in my repertoire to decipher is group size, so bottom middle.

Please tell me why I'm wrong....

[DaveMc]

Thanks Jason for answering so quickly and that is one of the answers I would expect and most people would agree.

I do not want to give away anything on the load yet. They are in order 1-3 across top and 4-6 across bottom.

perhaps a better way to then ask is assume the above but answer in two possible scenarios

Possible scenario 1: Testing random components in random order (e.g. primer brands or seating depth)

Possible scenario 2: We are testing powder charge and going up from 1-6.

[Julian D]

My uneducated opinion would be target 4 , only because it looks more like a peak rather than a trough.

[jasmay]

With that info, I would still lean towards target 5, 4 has a high POI and the POI declines across 4,5&6 consecutively, 6 has me a little concerned, but with the outlier sitting above the assumed group it is preferable to target 3 where it is dropping out the bottom of the group. the group is just starting to tigheten at 3, but we don't see it truly tighten until 4, 1 & 2 are open groups and appear to be useless to me.

Dave, this is exactly the type of training we need n this forum, and I applaud you for offering it!!

[Nathan P]

I'm assuming your testing different loads and are yet to fine tune seating depth, soo would say target 4 because it looks to be at more of a peak and just needs some seating depth fine tuning to bring in the out lying shots

[plumbs7]

Without any wind I'm with Julian 4 !

[mike H]

I am suspicious,it could be the same load in each target.
Mike.

[DaveMc]

Spot on Mike

And yes there is simulated wind (fairly reasonable conditions or good wind reading though)

just to point out I wasn't trying to trap anyone there rather less demonstrate what we all would tend to do. No one is alone in thinking a couple of these groups were outstanding. In fact some big brand hunting companies claim their rifles are "sub moa" based on 3 shot groups. In this case 2 of these groups are sub quarter minute at 1000 yards. Pretty impressive huh? Here is a 5 and 10 shot example - They suffer the same issues.

ALL THESE GROUPS ARE SIMULATED FROM THE SAME RIFLE - This is just the way shots fall by pure coincidence.

What we are doing is essentially sampling from a population. The population being all the shots that rifle/load combo can shoot and our sample is the group or series of velocity measurements or anything else we take. It is like grabbing people randomly from the street and trying to measure average weight of the population. The first 3 people you meet might be large adults and therefore not representative at all - in fact your results could have a low variation (and tight SD) but be way off the average/mean/mode or median. Other times you will get a 1 year old baby and 2 kids at school, similarly you will get 1 kid and two adults etc etc.

I will come back later and discuss in more detail, what we should be doing and how to interpret results.

5shotgroups.jpg

10shotgroups.jpg

[DaveMc]

So we have had little chance to look at these. I will get this model finished and a link somehow. Pressing F9 to recalc is mesmerising to watch the randomness of groups. The story however can be seen in these samples I have posted.

SO - what do we take out of this. I will leave it for another day or two to discuss further but just point out now to the "great groups" in every case - including the lovely 1000 yard 60.9 (10 shot group) with a pretty "average" rifle. This would be a great 1000 yard heavy BR group too.

All groups and velocity measurements we take will have smaller extreme spreads than the actual population of shots the rifle will produce. We are often "fooled" (for want of a better term) into thinking we have something great. And we all (as human beings it is our nature) keep quoting that magnificent 60.9 ("so I KNOW my rifle is accurate - it must be the range or the targets or the scope etc etc".).

Once again the words we all need to tell each other and ourselves is "thats great - now repeat it." repeatable SD or group size is the important factor. "agging" in BR rather than one off groups.

This is also a big reason why many people do not believe in velocity and group testing but rather just like to prove it in the paddock.

[DaveMc]

OK - I have been at it again (sorry! ).
In order to try and give a bit more insight into these sampling errors I decided to run another monte carlo test to generate some numbers on velocity testing.

What this test does is generate 9,990 random velocities around a mean of 3000 fps and a standard deviation of 5 (can be adjusted in the model). This effectively creates a "population" of shots with a known standard deviation of 5. I then broke up this population into 30, 15, 10, 5 and 3 shot groups and measured the standard deviations and extreme spreads of the smaller groups.
Attached is the table of one set of results (hitting F9 recalcs and generates new set of numbers but the results are generally very similar)

sdinsight.jpg

[DaveMc]

Interpretting the results:

what these figures mean??? -

The max and min figures down the bottom are the largest and smallest standard deviations calculated from the respective sample sizes. ie from the 3330 * 30 shot samples the lowest standard deviation was 3.23 and highest was 6.85 (population is 5) and 75% of samples were between 4.14 and 5.69.

5 shots ranged from 0.50 to 10.89 and 75% were between 2.74 and 6.73

Don't take too much notice of 3 shots - you can calculate mathemetically the sd of 3 shots but it means very little - I have it in therte mainly for the ES calculations to follow.

[plumbs7]

Dave , is an SD of 2 even achievable over a large population of shots ? It maybe achievable over 5-10 shots but as you say in a large group of let's say 100 shots even temp change would change the spread!?? I maybe chasing rainbows for not much?

[DaveMc]

No Graham, It is not likely achievable. I saw KH refer to a comment about "agg" of groups or velocity spreads on another thread and I am about to present some information on that. This is a term you will often hear amongst bench rest shooters and is the best way we can get a true picture of what is happening with a rifle.

We as shooters cannot do 30 shot groups without burning out our barrels, and certainly 100 shot strings are going to kill them and have other effects on velocity spreads etc.

"Agging" or taking the average of many smaller groups I am about to demonstrate is nearly as statistically powerful as one big sample size and in our case a LOT MORE likely to give you a good representation of what is going on. I was working on the spreadsheet model last night and got the results I was trying to demonstrate but I will need to get home to post. Normally I would send through to Peter for some scientific appraisal before putting out there and will try and get in there with a laptop for his comments first.

What I am happy to say though is whilst individual groups or sets of velocity measurements should not be considered with too much rigour. The combined average of several is statistically very powerful.

Even 3 shot groups surprised me on the Monte Carlo results as to how good the averages of many smaller groups were at estimating the true standard deviation of the population.

What will come out of all this flurry on ES/SD, sampling and other is some clearer understanding of what is going on and how to sample/load develop with some meaning and direction. I see a lot of frustration when things aren't repeatable (ie one or two good groups/ velocity sd's followed by pain). I think the term "chasing rainbows" should be twisted around for this purpose. Many people chase rainbows (or more importantly chase something they have seen once in testing when it really isn't there - they end up going around and around in circles) without clear direction and have been mislead by not understanding the results. It can become a frustrating waste of time effort and money.

I am going to quote KH from the "what is required to win" thread in the general forum as I think it is spot on and comes from years of experience

"I would agree 100% with you Richard. Unrealistic velocities, unrealistic SD figures & unrealistic group sizes. Sample size has been mentioned on this board, this needs to be taken into consideration when analysing data. Very often we will latch onto what we want to see, enter that word "unrealistic" again. A small 5 shot group does not mean that your rifle is agging that group size. A small string in a velocity check giving a low SD does not mean all strings will have the same low SD. So we take all these as "indicators" & test on the range in club shoots to prove performance consistency. One small group with low SD means nothing if the rifle can't consistently produce those results. Many times a small sacrifice in group size & or a small sacrifice in SD will produce a more reliable result overall. Thats a bit longwinded for me "phew".
Keith H."

What can be achieved and what you need to achieve is purely subjective. What are your goals and are they realistic? There are some truly outstanding freaks of nature out there with regards to 1000 yard elevation and velocity spreads but are these achievable for most?? Are they realistic for the person, barrel and gear. More importantly do you need that level of accuracy to win?? - Each individual needs to make those decisions and determine how far they want to go.

HOWEVER - With improvements in components, knowledge and barrel quality (Not to mention the all important quality of the smithing work) we are starting to see more and more people achieve outstanding things.