Google BigQuery: 7 Fascinating Facts

Over the last year or so, we have worked extensively with Google Cloud’s premier offering, BigQuery. BigQuery was built because no traditional technologies at the time could perform fast enough to support Google Maps. It will be a key component of cloud business intelligence and big data solutions very soon. As a super fast API that works as an analytical database, BigQuery is such a different animal that we can’t help but continue to be fascinated with it.

7 fascinating things about Google BigQuery:

1. Ingests while serving data

BigQuery is read-only like other analytical databases. However,what differentiates it is that it can be fed data at the same time that it is ingesting data into a database. As such, multiple partitions are not needed. 

2. Auto-optimization

Contributing to its mesmerizing speed is auto-optimization. BigQuery does not require the constant maintenance of indexes, as it stores data in a columnar-like structure. This makes processing data shockingly fast.  It is amazing how it “just works”.

3. No size limits

The database size has no limit.  This means that it can be as big as you need, which is just unheard of. We have tried this and it can easily handle terabytes of data. You can store all the data your business needs without impacting performance. This also means there are no servers or hard drives to manage.

4. SQL-like queries, easy to use/adopt

BigQuery uses a SQL-like query syntax? Yes. It is as easy to use as writing a simple select query. Given that SQL is widely used, this will open the door for more people to interact with BigQuery.

5: Ease of management

A simple cloud console allows you to manage the database objects like tables and views, but most important, you can secure the data assets within the data sets. Jobs history allows you to manage database updates, status, and/or errors.

6. Nested json, support for complex schemas

Don’t be fooled in thinking that something so fast can only handle extremely simple schemas. It actually supports nested json. BigQuery allows ingestion of the most complex structures, that are exposed via today's common web services. 

7. Super fast

It can analyze billions of records in seconds, not minutes or hours or days like other databases. 

Bimotics automates the ingestion process further making Google BigQuery and Bimotics a great combination. If this sounds too good to be true, reach out to Bimotics.  In just a few minutes, we will be happy to show you a demo of these amazing capabilities! 

Bimotics looks forward to 2016

In our year end post, we revisited some of our big wins in 2014. The beginning of a new year means goal setting and planning for what is to come. A new season means new opportunities and we are full of optimism!  This blog entry focuses on those top goals for our startup. We are sure that the next 12 months will not be without pivots and turns, but this is just part of the journey we take and we are looking forward to the challenges ahead.

SMART Goal: 1 new product and brand launch this year 

After much debate, we are considering changing our name to something equally impactful but easier to pronounce. When we chose Bimotics years ago, we were lucky to get a service mark and .com domain fairly easily. Both items can be hard to come by for a startup, especially since our name has a  uniqueness of meaning.  As we began pitching  and discussing our company with people, we realized that “Bimotics” is tough for people to pronounce and read under pressure (especially while introducing our startup in front of a crowd). We are currently reviewing a short list of names and icons that meaningfully capture the essence of our core business as well as the name Bimotics does.

SMART Goal: Roll out to 2 core charting capabilities with ability to measure usage

At long last, we are excited to  roll out our end-user presentation layer. We have heard it time and again from both prospects and advisors that UI sells.  Our experiences show that people and customers expect a solid foundation and for the “plumbing to work” so to speak, but the way they differentiate the value of things is based heavily on visualizations. So while we consider the 2014 launch of our big data tool marvin. a great milestone, we need to take this robust foundation and plumbing for analytics solutions a step further. In 2015, we will be rolling out a visualization layer that allows non-technical users to interact with Big Data easily and affordably and peek into their data in a way that they never have before. We believe funneling more capital towards front end development is what is needed to demo our true value while increasing revenue.

SMART Goal: Participate in  2 tech community events a month

We are going to help grow our local tech community. Bimotics has attended key meetups within South Florida in the past but this year we aim to increase our involvement. By being participants, organizers, audience members, and sponsors over the past few years, we have increased our understanding of the value a community of tech entrepreneurs can bring. Whether it was introductions to angel investors or the identification of helpful resources for product development, each event benefitted us in some way. We also have something to give back in terms of experience, expertise, and lessons learned along the road and we look forward to sharing our knowledge with the community in 2015.

So it is going to be another busy year at Bimotics. Of course there are several other more tactical goals for the year, but from a company perspective, we look forward to working to meet these goals in particular. None of these will be easy nor can be done individually. Let’s get to work team!

Fireside Chat with the Creator of marvin.

This month, you have been hearing a lot about the marvin. product launch.  Here at Bimotics, it is a very exciting albeit dramatic time.  We are thrilled to see how the whole team has come together to carry out all the tasks and steps needed for launch. Synergy is real. We are seeing it first hand. 

To get a better sense of what we are bringing to market here is the transcript of a question and answer with our technical lead and co-founder, Roberto Landrau.

How did marvin. come about? 

While developing our big data and business intelligence tool for small and medium sized businesses, in almost every instance we needed to import multiple files from multiple APIs, from multiple clients, during multiple time frames.  The process was extremely tedious if not impossible without some sort of intelligent and automated ingestion engine that could get a handle on all this data.  From this quandary, marvin. was born.

marvin. is super easy to configure and allows users to manage Google Cloud APIs like Google Cloud Storage and BigQuery. Before marvin., files could only be entered into Google BigQuery one at a time manually --great for testing, but not feasible for our SMB analytics product.  Initially, we decided to build marvin. for internal purposes in order to manage the ingestion of our clients’ data.  Almost immediately, some of our more sophisticated clients asked to use marvin. for their own Google Cloud Platform needs. We decided to make it a commercial ready product by adding few additional capabilities such as multi-user, account based settings, support model and subscription based payments.

What makes marvin. a cloud BI tool? 

marvin. was born in the cloud, it scales to hundreds if not thousands of users seamlessly by leveraging Google’s App Engine. marvin. is a cloud BI tool as it ingests data from on-premise files or cloud files into one of the most powerful analytical databases on the market. Currently, marvin. focuses on the ingestion of files, but in the near future our gallery of API connectors and the integration of our visualization engine will allow users to store massive amounts of data as well as analyze and manipulate it simply through a robust graphical interface.  

What makes it for big data?

marvin. is capable of ingesting data for multiple clients at the same time, and its storage capacity is  limitless. There is no need to manage servers, or disks, or IT resources. Users can analyze billions of records within seconds, not days or weeks. Watching marvin. in action is truly impressive.  With its parallel import capability and ease of use, it is the perfect tool for your big data needs. Also consider that SMBs can use this tool to gather data in real time from multiple applications commonly used to support day to day business, such as popular accounting and CRM software. Our ingestion engine also supports data in more complex formats like nested JSON, allowing businesses to overcome both big and small data challenges from a single location.

Who do you expect will use it?

marvin. is primarily geared towards Google Developers and Google Cloud Platform users. Non-developers that still work in with technology and IT will rejoice as there is nothing to code. Simply gather your data in CSV or JSON, upload the file and configure an automated import. Subsequent files will follow the same pattern.

Users with voluminous amounts of data and no time to deal with servers, disks and networks will benefit the most from marvin.. The UI is easy to use and wizard setup screen helps you to configure the ingestion within seconds.

Where can I find marvin. on sale?

marvin. is now available via the Google Web Chrome Store or at www.bimotics.com/marvin.  Be sure to check out our animated overview video as well!

What’s with the lowercase m and the period?

Bimotics is very serious about the products we build, but our marketing team is also focused on one of the cornerstones of our company culture: fun.  We certainly had a blast coming up with marvin..  marvin. stands for Massive Analytics Repository on a Very Intelligent Network, quite a name for a cute little robot.  By going with a small “m” and the period at the end, the name transforms into an iconic symbol, a bold statement.  Once you get to know marvin., you will surely love him. period.

So What is This Smart Data?

The last few months, I have noticed the term “smart data” popping up in sessions and blogs mostly related to big data and data science. The content is full of the value of using smart data to answer questions, but what exactly is it and how does it differ from the regular data we collect everyday? I did a little research and here are some things that I have learned. It turns out the concept of data marts and analytics marts that Bimoticsdiscussed  earlier this year are very similar.

According to Cambridge Sematics, a company specializing data and analysis, describes smart data as a set of data that has been collected in such a way that it can be optimized discovered, integrated, visualized, and analyzed. There are technology vendors that have described smart data as big data analytics or analytics on top of big data.

Smart data usually is collected from many different sources and applications. The data is then aggregated into a single location. While bringing it in, the different data elements need to be organized and standardized so that they relate to one another in a way that represents reality. For example, an order has many product line items and each product has a price. While this data modeling reflects how all data for business intelligence starts out, there are nuances that eventually makes this data “smart”.

To start, smart data is not a just a collection of all the data related to a particular subject. Instead, the information and data elements are evaluated for significance and the ones that are highly related to the subject are kept in the set. This means that the smart data model needs to be flexible so that data can be included and excluded as needed. I have seen this concept of smart data in articles related to customer insights and predictive analytics. Identifying and keeping only the most meaningful data points is at the heart of these kinds of analyses. 

The relationships between elements are defined and form a common set of terminologies, so that these sets becomes more understandable. Maintaining the meaning of the data and making sure that it has not changed from the original is key and often complex. The effort pays off when the number of false positives decline as well as when the data begins to be more  purposeful and can be used in many different ways to  answer many different sets of questions.

While, I am just beginning to understand the concept of smart data I am eager to learn more as this concept emerges into something as popular and common as big data.

How Business Intelligence will evolve in 2015

If you are in the enterprise technology industry, you probably know that innovation within the space can be a slow process. Companies that have invested many millions of dollars and thousands of hours in legacy technologies are often reluctant to make bold changes. While it may feel like innovation in enterprise analytics moves at  crawl pace, each year is not without its advances. Today we focus on the progress we predict to see this year in the space for business intelligence, analytics and big data technology.

Greater demand for sensor data analysis

With the onset of the Internet of Things, demand for the analytics and reporting of sensor data will continue to take off.  The use of sensors to take measurements and monitor activity has already skyrocketed.  Just as fitness trackers have managed to put another dashboard into your daily view, the need for solutions that process and display information from sensors can only grow. Even in industries such as commercial retail, the use of sensors like e-beacons, smart shelves, and intelligent fitting rooms are taking off and adding to the demand for systems that can handle big data sets and charts.  This sensor technology and its demand for data analytics is still in its infancy.

More mobile user interface solutions

At Bimotics, we are increasingly getting requests to be able to provide our dashboard on mobile devices.  With mobile phone screen sizes growing and users’ mobile savviness increasing, business intelligence and analytics will continue the evolution of dashboards on phones and tablets. I expect to see  sophisticated dashboard-applications take off,  building on the currently available charting user interfaces that have adopted responsive design.

Naming the flavors of business intelligence

This trend became more apparent last year. With the  exponential growth of data and its uses, I expect a further  “labeling of the flavors” in 2015. Big Data came first for nomenclature to distinguish large volumes of data from disparate sources. Smart Dataemerged as a method of using data that matters and using that data to build foundations for predictive solutions. Terms such as Dark Data, Operational Analytics, and Text Analytics are making their way into the vernacular. The labeling of the different flavors of business intelligence solutions will continue as this large and diverse industry becomes more mature, granular, and specialized in focus.

Even if innovation and advancement takes more than a year for full adoption to occur, things do change from year to year within the business intelligence and analytics industry. Looking beyond purely technical strides and advancements, these are the three broad trends I I expect to see unfold in 2015.

Big Data Analytics: Marketing Came First. Why Engineering Is Next.

Big data solutions have made it.  It is clear that business intelligence and advanced analytics solutions have been embraced by the market and that adoption and innovation continue their upward trajectory.  Many of the first companies to provide big data solutions were marketing-focused companies specializing in predictive analytics.  As this market matures and marketing companies find their niches, the conversation and question naturally moves to “who is next?”.  What other enterprise group is ready to evolve and innovate big data analytics?

It does not appear to be sales despite early advancements in the analytics space. Perhaps because sales analytics investments were made early on and the benefits from those initial investments are still being reaped.

Customer service looks to be a likely candidate however budgets do tend to be tight for this group.

Our educated prediction here at Bimotics is that the next boom in big data analytics will be in the engineering space.  Here is why:

Systems or application logs: in order to provide real time monitoring, tech support and proactive maintenance, applications generate millions of rows of data, describing all kinds of processes.

Application Usage Analytics: customer experience and human factors are based on usage analytics. This means applications capture the flow and interactions of users in terms of features they use, time on the app or pages, and patterns of usage.

Metadata Management: Data traceability is of the utmost importance and "data about data", ie metadata, is a huge arena. Payload of data transfer is also considered in this category where sometimes the metadata generates more data than the payload itself.

Cohort Analysis: By developing A/B testing, engineers can improve conversion ratios or better UX / IU to increase usage. Cohort analysis uses two groups as a benchmark. This generalization of groups requires more data storage.

The big benefit of engineering-related data analytics is that insights and performance directly impact the products and services which are core to the business. Understanding how customers use your application in particular and what makes them tick can make product planning and prioritizing features more methodical and effective. This is why we believe that engineering analytics will be the next big wave in the sea of Big Data Analytics.

Building an Online Data Warehouse Part 2

Google Cloud Platform is a great option for companies looking to build a Data Warehouse on the cloud.  As described in Part 1, you have two options: hire a developer to code and integrate on top of those great APIs, or use tools like marvin. to help you automate some of the processes that are required to build an online data warehouse. In Part 2 and 3,  we will discuss the most valuable sections of a Data Warehouse, Analyzing and Visualizing your Data.

Analyzing the Data

marvin.’s super powers come from Google BigQuery API, an analytical database on the cloud. marvin. automates the process of inserting and storing data uploaded in the previous step. Together, Google BigQuery and marvin. are the analytical engine for your data warehouse. Your new Data Warehouse is limitless in terms of size and analytical computing power. Even better, there is nothing to code, no servers to manage, and no hard drives to setup. To learn more check out our blog:  Google BigQuery: 7 fascinating facts.

With marvin., you create datasets, make tables that host data, and preview that data in a table format. The data is no longer raw. Instead, your precious data is stored like in a huge spreadsheet that can be analyzed very quickly. Think billions of records in seconds. Think Big Data. Think cloud bi.

What if your business only has a few thousand records? No worries. marvin. will manage small data just as well. After all, a small business does not need to analyze a billion records of its data to make informed decisions and leverage insight to compete with other businesses.

Your new analytical engine also allows users to interact with simple SQL statements. If your team does not know SQL, it is also compatible with multiple visualization engines like Tableau and BIME, popular programs that present your data graphically. Visualizing the data is the last piece of the 4 steps of the simplified process.  We will explore data visualization in Part 3.   

Google BigQuery is a Big Deal : A Brief History of Database Evolution

As a person so close to, active in, and passionate about the technology industry, I love sharing my knowledge with non-technical crowds from time to time.  It is rewarding and refreshing to simplify some difficult concepts, take a look back at history and gain new perspective.  I recently found myself educating the less-technical side of our team at Bimotics about the incredible power of analytical databases like Google BigQuery. What I realized is that folks can better appreciate BigQuery’s  capabilities by  first understanding how databases have evolved over time.  The following is a short evolutionary history of databases and a summary of why BigQuery really is a big deal!

Writing

Most of the databases since the 1970s have been developed, designed and architected with the main purpose of writing and capturing data. Most of these "writing" databases are based on storing data in rows, as this makes it very simple to gather, append and collect data as quickly as possible. SQL became a very common language to inquire or query the data from these writing-optimized relational database management systems. As the volume of data has grown exponentially over the last 40 years, these writing-optimized databases have started to show their age in terms of analyzing large numbers of records, making analytics on top of these architectures time consuming and slow. Big data has also been focused on collecting everything, further increasing the size of and strain on these write databases.

Reading

Although the challenge of creating analytical databases or databases optimized for “reading” also started in the 1970s, it was not until the 1990s that reading-optimized database adoption  began to grow and mature. Multiple approaches like Cubes and non-SQL languages like MDX that focused on being able to read row based (“write”) databases more quickly, usually required processing or ingestion that took time and introduced latency. These approaches also made it hard for analysts to leverage “read” databases like HP’s Vertica, Cognos, Hyperion or Microsoft. Eventually, “reading” databases like SybaseIQ (1995) were developed and based on storing data in columns rather than  rows. This design optimized the capabilities to query and read information quickly. The next step was to move these analytical databases to the cloud.

Why Google BigQuery is a big deal

It was not until the mid-2000s that cloud databases based on columnar storage with SQL language compatibility became a real solution for analysts. Google BigQuery made it to market by 2010, allowing for unparalleled speed regardless of size. Based on a Dremel whitepaper from 2006, the reading optimized database has unlimited growth and unparallel performance. BigQuery is a big big deal in terms of analytical databases because it allows analysts to query their data very quickly using SQL regardless of the size. You can ingest data very fast, while never stopping to query thus removing the latency introduced by early analytical databases. BigQuery is available as a cloud service that does not require configuration or hardware setup.  It is auto optimized with no indexes to manage. Finally, it is fully secure and capable of complex nested structures to support web data. BigQuery is indeed a big deal and the product of 40 years of database evolution!