Big Data Analytics Consulting

[RussianMike]

Hi everyone!! I am new to these forums, glad to be here in the Statistics community!

So me and a friend of mine are planning to create a Data Analytics startup. What we want to do is try to target small to medium sized business. We want to see if we can get any kind of data from these businesses (pricing data of their products, their inventory data, and any other kind of data) and try to analyze it using the various applicable statistical methodologies in order to come up with statistical results that can used to either decrease the costs or increase revenues for the business. We are trying to find other people who would be possibly interested in joining us in this. My friend is the stronger of the two of us in the Statistics/Mathematical sense. He has a B.S in Physics from Caltech and a Masters in Mathematical Finance, I love math and statistics but am not as technical as him. I wanted to see both other people's opinions on our desired project? Can anyone provide feedback? Also, if anyone is interested in exploring the idea with us feel free to comment here or reach out to me. Thank you

 

[RussianMike]

Can somebody please provide some sort of comments, I thought people would have multiple responses by now. Can someone please tell me about their insight in this. I would really like to know people's opinions as to whether they think this can work or not, etc??

 

[mlangosch]

I have worked for both a very large company and now a small one(but with big revenue). The thing that I have found is that neither one is in a hurry to share their data and they are constantly worried about security. While I think a lot of business could benefit from analytical services, they will most likely be hard to convince.

 

[spunky]

i think i have to agree with mlangosch.

my husband owns a couple of record labels in canada and latin america with distribution cross north america and the big south american economies (e.g. Brazil, Argentina, etc.) and i asked him what he thought about outsourcing some big data analytics for his business (not the first time i've asked him because although i am proficient in stats i'm not proficient in the kind of stats needed for Big Data). his answer was along the lines of "you think i'm gonna share THAT info with some random stranger?"

ON THE OTHER HAND THOUGH just from watching the news and talking to people in business i know if you can do Big Data proficiently companies are gonna fight over for you like crazy

 

[noetsi]

I think the odds of most corporations sharing data with analyst outside the company are about the same as corporate executives asking for a pay cut.

I also think you will find few corporations who really want involved statistical analysis (if they want data analysis it is likely to be financial not statistical and the two are very different).

I saw this first hand working for a mid size, but tech savy, insurance firm.

 

[spunky]

I also think you will find few corporations who really want involved statistical analysis (if they want data analysis it is likely to be financial not statistical and the two are very different).

this i don't quite agree with. Big Data analytics is not just "statistics". it involves database management, internet traffic flow, etc. it can have a financial aspect, a market aspect, a social science aspect. with the constant push of society towards digitalisation there really is a lot of information out there that corporations and gov'ts want to harvest and process.

 

[noetsi]

Perhaps, but I think you will find formal statistical analysis in private firms vanishingly small today with a fair amount of financial modeling (done by actuarials).

 

[spunky]

Perhaps, but I think you will find formal statistical analysis in private firms vanishingly small today with a fair amount of financial modeling (done by actuarials).

well... that does depend on what you mean by "private firms". i, for once, could not even imagine any big pharmaceutical company getting rid of its research/clinical trials section.

and i'm not sure what you meant by the 2nd comment. are you implying that actuaries are taking over from data analysts or that both data analysts and actuaries are becoming irrelevant, in your own opinion?

 

[RussianMike]

Yea that's something I am concerned about as well, that small business is going to be concerned about data privacy. But than how does big data analytics work at large corporations? I mean if any of say the Fortune 500 companies have outside consultants working on their big data, they are doing the exact same thing, giving out their data to the consultants right? The only difference is those are mega corporations whereas I am talking about small business. Or am I missing something here?

So what do you guys think about taking say a grocery store and doing statistical analysis on it. If we can say get data on pricing of every single item in the store by date and time its sold, all the data on inventory (how much of each item is in the store to see how long its there for) and also say create some sort of variable for the distribution of different food items in the store (to check if the location within the store is going to impact how much people buy it) and run all sorts of statistical analysis on these data to determine if the optimal amount of all food types is sold at the optimal price? (Say should this meet be sold at $2, $4 per pound, aka what is the optimal price, how much of it should we have in stock?) What is the best way to layout the different products in the store according to data analysis? What do you guys think of this general idea? Is that something small or midsized business would have interest in?

 

[noetsi]

and i'm not sure what you meant by the 2nd comment. are you implying that actuaries are taking over from data analysts or that both data analysts and actuaries are becoming irrelevant, in your own opinion?

Essentially the former, or more accurately I mean with some rare exceptions such as medical firms, few businesses have data analyst who do statistics.

Admitedly I have never seen any formal analysis of what percent of business employ statisticians. This is only my opinion

 

[noetsi]

But than how does big data analytics work at large corporations? I mean if any of say the Fortune 500 companies have outside consultants working on their big data, they are doing the exact same thing, giving out their data to the consultants right?

You are talking about a tiny number of firms there. They have the resources to vet the individuals and firms they employ who get their data (and those firms will generally have security systems in place as well). So its a totally different reality than that faced by most firms including quite large ones.

There are specific analystical models that exist for the movement and storage of goods and services (what business call logistics not to be confused with logistic regression). With speciality software that handles this. I think you will find that firms interested in this hire experts in such software (who will not be statisticians normally) to address that.

 

[spunky]

Essentially the former

OMG noetsi... now, what *EXACTLY* do you think actuaries study? IT'S.ALL.DARN.PROBABILITY (with financial math... which, if you look at it closely, has A LOT of statistics). actuaries are applied STATISTICIANS. actuarial sciences are basically statistics used to asses risk and risk management. i know this for a fact because i wanted to become one. have you had a look at the certification exams you need to take to become an actuary? one of them is pure probability, plain and simple. or have a look at the courses one has to take to become an actuary. you become a statistician (and a rather good one, i must say) as a by-product of becoming an actuary.

you seem to have this incredibly weird misunderstanding of who statisticians are and what they do biostatisticians, econometricians, financial modelers, the quants from Wall-Street, marketing research, psychometricians. it is ALL statistics! they have just been refined to adapt to the specific that each area needs.

 

[noetsi]

Have you ever asked acturials whether they do statistics (as compared to financial analysis)? None that I ever worked with ever brought up statistics - they did math and finance not stats

I don't agree that financial analysis is stats (certainly as we use it on this board commonly). Finance and stats are both math based but they are very different realms of math and commonly people who do financial analysis don't do much stats and vice versa. My brother - a brillant financial analyst who made lots of money doing that for companies - probably never took a stats class in his life. And I believe that is common of financial analyst. They know finance, the flow of money, etc not statistics.

That is my experience and all I can comment on. Obviously you have had other experiences.

 

[spunky]

Have you ever asked acturials whether they do statistics (as compared to financial analysis)? None that I ever worked with ever brought up statistics - they did math and finance not stats

not only have i asked them. i *STUDIED* actuarial sciences. i took courses in it, and i'm well aware of the mathematical and statistical requirements on them. and i know it's just the fact that they are so specialised in the area of risk management, that a lot of the statistics and mathematics relevant to actuaries tend to stay within their confines.

and what, in your opinion is the difference between financial mathematics and statistics? from just look at this very specific examples from the wikipedia article on mathematical finance:

http://en.wikipedia.org/wiki/Mathematical_finance

"Quantitative derivatives pricing was initiated by Louis Bachelier in The Theory of Speculation (published 1900), with the introduction of the most basic and most influential of processes, the Brownian motion, and its applications to the pricing of options. Bachelier modeled the time series of changes in the logarithm of stock prices as a random walk in which the short-term changes had a finite variance. This causes longer-term changes to follow a Gaussian distribution. Bachelier's work, however, was largely unknown outside academia."

"the fundamental theorem of asset pricing by Harrison and Pliska (1981), according to which the suitably normalized current price P0 of a security is arbitrage-free, and thus truly fair, only if there exists a stochastic process Pt with constant expected value which describes its future evolution: A process satisfying (1) is called a "martingale". A martingale does not reward risk. Thus the probability of the normalized security price process is called "risk-neutral" and is typically denoted by the blackboard font letter "

everything i highlighted comes from statitics. now, show me ONE instance of mathematical financial theory where statistics is not involved. just ONE and even if it's not explicitly there i can guarentee you i can link it to something in statistics because finance is not an exact science. there is too much uncertainty involved in it and wherever uncertainty rears its head, statistics is present

 

[bryangoodrich]

Noetsi, your argument amounts to "businesses don't hire statisticians; statisticians are being replaced by actuaries." First off, no, actuaries are far from the only replacement. There's data analysts, business analysts, and many other analytical positions out there. Secondly, I can agree that firms scarcely hire statisticians, because "statistician" as a job is very narrow. The same applies to mathematicians. The mathematical sciences provide a skill set, a way of thinking. That doesn't mean people that go through a statistics program and get hired in business aren't finding jobs or that they're only finding actuarial jobs. If your argument has any salience, it would confirm that hypothesis. However, from experience, you're just wrong. What do you think the craze in data scientists the past couple of years has been about? It isn't financial analysis or risk management. It's data, and the means to analyze that data. That is the confluence of computer and mathematical sciences. That is what makes statistics sexy.

 

[noetsi]

All I can say byrangoodrich is that has not been my experience nor of those that I knew, who worked in data analysis at a pretty high level. Since I have never seen anything on this topic that is formal, that is surveys of what businesses do and do not do in these areas, all I have is my experience and those that I have talked with. In every organization I have worked in, and they have been reasonably large and in some cases very data driven, I was the only one doing any statistics. And obviously my statistical knowledge is not all that awesome so we are talking low lying fruit

Spunky your experience with acturials obviously differs from mine. When I worked with a group of them in the insurance industry they did no statistical analysis. Obviously different groups vary on this. I will keep that in mind.

 

[bryangoodrich]

I find it hard to believe that you were the only one doing statistics in a large organization. Maybe you were the only one you knew was doing it, but like most organizations, units stay to their silos. It's one of the most common talking points in the webinars I've been watching the past several months on data science and analytics: removing silos, both in terms of data and talent, being transparent, making data available, etc. There is friction for companies to move to a data-driven culture, especially one outside of IT that gets utilized by analysts, and that's because the right talent isn't ideally positioned, that data is horded, that the infrastructure lacks investment, and so forth. It doesn't mean there aren't analysts. Companies on top of their game, like Facebook and Google, specialize in making analytics integrated into every area of their business. Financial, retail, and telecoms are highly invested in customer analytics, precisely because it's those statisticians that are helping them understand their customers. Market specialists, economists, and the like are statisticians. I'm pretty confident there is data out there for where graduates in certain majors end up.

 

[mlangosch]

I was hired into my position because of my mathematics background. However, the company was considering hiring a part-time person to model our data and take a look at a current transportation logistics algorithm and recommend improvements, I just got lucky as far as timing was concerned! There is a niche out there(our company occupies a small one), it's just a matter of figuring out a good way to market it and prove your abilities and results. Maybe look into targeting a particular sector of small business, like you mentioned grocery stores, and develop an easily adaptable system that uses somewhat universally available data(i.e. every store would have it available, but not necessarily public).

 

[noetsi]

I find it hard to believe that you were the only one doing statistics in a large organization.

I guess it depends on what you mean by large. In the last two cases it was one over a thousand employees which is large compared to the norm in the US.

It would be fasinating to know how many companies with over a thousand employees do employ people with statistical training. I was doing statistics long before I had a formal degree in it, which reflects a basic problem with gathering such data. It is fasinating that different individuals come to such totally different conclusions. I hope I am wrong about the state of statistics in the US.

 

[RussianMike]

Mlangosch, can you further expand on your idea please. Let what specifically do you suggest say in the grocery store example?

Also, for the ongoing debate here about actuaries, even though that is kind of aside from the subject of this thread, my best friend who I see on an almost daily basis is basically an actuary/quant. His degree was in actuarial science and he past all the exams in the finance/investments track quickly moved up the ladder to the VP level really quickly. He basically models all sorts of investment vehicles, etc using elements of both math and statistics. I dont think the quant world is pure math neither is it pure statistics, its a combination of both. you need knowledge of calculus and you need to know statistics as well.