Google BigQuery: 7 Fascinating Facts

Over the last year or so, we have worked extensively with Google Cloud’s premier offering, BigQuery. BigQuery was built because no traditional technologies at the time could perform fast enough to support Google Maps. It will be a key component of cloud business intelligence and big data solutions very soon. As a super fast API that works as an analytical database, BigQuery is such a different animal that we can’t help but continue to be fascinated with it.

7 fascinating things about Google BigQuery:

1. Ingests while serving data

BigQuery is read-only like other analytical databases. However,what differentiates it is that it can be fed data at the same time that it is ingesting data into a database. As such, multiple partitions are not needed. 

2. Auto-optimization

Contributing to its mesmerizing speed is auto-optimization. BigQuery does not require the constant maintenance of indexes, as it stores data in a columnar-like structure. This makes processing data shockingly fast.  It is amazing how it “just works”.

3. No size limits

The database size has no limit.  This means that it can be as big as you need, which is just unheard of. We have tried this and it can easily handle terabytes of data. You can store all the data your business needs without impacting performance. This also means there are no servers or hard drives to manage.

4. SQL-like queries, easy to use/adopt

BigQuery uses a SQL-like query syntax? Yes. It is as easy to use as writing a simple select query. Given that SQL is widely used, this will open the door for more people to interact with BigQuery.

5: Ease of management

A simple cloud console allows you to manage the database objects like tables and views, but most important, you can secure the data assets within the data sets. Jobs history allows you to manage database updates, status, and/or errors.

6. Nested json, support for complex schemas

Don’t be fooled in thinking that something so fast can only handle extremely simple schemas. It actually supports nested json. BigQuery allows ingestion of the most complex structures, that are exposed via today's common web services. 

7. Super fast

It can analyze billions of records in seconds, not minutes or hours or days like other databases. 

Bimotics automates the ingestion process further making Google BigQuery and Bimotics a great combination. If this sounds too good to be true, reach out to Bimotics.  In just a few minutes, we will be happy to show you a demo of these amazing capabilities! 

Fireside Chat with the Creator of marvin.

This month, you have been hearing a lot about the marvin. product launch.  Here at Bimotics, it is a very exciting albeit dramatic time.  We are thrilled to see how the whole team has come together to carry out all the tasks and steps needed for launch. Synergy is real. We are seeing it first hand. 

To get a better sense of what we are bringing to market here is the transcript of a question and answer with our technical lead and co-founder, Roberto Landrau.

How did marvin. come about? 

While developing our big data and business intelligence tool for small and medium sized businesses, in almost every instance we needed to import multiple files from multiple APIs, from multiple clients, during multiple time frames.  The process was extremely tedious if not impossible without some sort of intelligent and automated ingestion engine that could get a handle on all this data.  From this quandary, marvin. was born.

marvin. is super easy to configure and allows users to manage Google Cloud APIs like Google Cloud Storage and BigQuery. Before marvin., files could only be entered into Google BigQuery one at a time manually --great for testing, but not feasible for our SMB analytics product.  Initially, we decided to build marvin. for internal purposes in order to manage the ingestion of our clients’ data.  Almost immediately, some of our more sophisticated clients asked to use marvin. for their own Google Cloud Platform needs. We decided to make it a commercial ready product by adding few additional capabilities such as multi-user, account based settings, support model and subscription based payments.

What makes marvin. a cloud BI tool? 

marvin. was born in the cloud, it scales to hundreds if not thousands of users seamlessly by leveraging Google’s App Engine. marvin. is a cloud BI tool as it ingests data from on-premise files or cloud files into one of the most powerful analytical databases on the market. Currently, marvin. focuses on the ingestion of files, but in the near future our gallery of API connectors and the integration of our visualization engine will allow users to store massive amounts of data as well as analyze and manipulate it simply through a robust graphical interface.  

What makes it for big data?

marvin. is capable of ingesting data for multiple clients at the same time, and its storage capacity is  limitless. There is no need to manage servers, or disks, or IT resources. Users can analyze billions of records within seconds, not days or weeks. Watching marvin. in action is truly impressive.  With its parallel import capability and ease of use, it is the perfect tool for your big data needs. Also consider that SMBs can use this tool to gather data in real time from multiple applications commonly used to support day to day business, such as popular accounting and CRM software. Our ingestion engine also supports data in more complex formats like nested JSON, allowing businesses to overcome both big and small data challenges from a single location.

Who do you expect will use it?

marvin. is primarily geared towards Google Developers and Google Cloud Platform users. Non-developers that still work in with technology and IT will rejoice as there is nothing to code. Simply gather your data in CSV or JSON, upload the file and configure an automated import. Subsequent files will follow the same pattern.

Users with voluminous amounts of data and no time to deal with servers, disks and networks will benefit the most from marvin.. The UI is easy to use and wizard setup screen helps you to configure the ingestion within seconds.

Where can I find marvin. on sale?

marvin. is now available via the Google Web Chrome Store or at www.bimotics.com/marvin.  Be sure to check out our animated overview video as well!

What’s with the lowercase m and the period?

Bimotics is very serious about the products we build, but our marketing team is also focused on one of the cornerstones of our company culture: fun.  We certainly had a blast coming up with marvin..  marvin. stands for Massive Analytics Repository on a Very Intelligent Network, quite a name for a cute little robot.  By going with a small “m” and the period at the end, the name transforms into an iconic symbol, a bold statement.  Once you get to know marvin., you will surely love him. period.

Building an Online Data Warehouse Part 2

Google Cloud Platform is a great option for companies looking to build a Data Warehouse on the cloud.  As described in Part 1, you have two options: hire a developer to code and integrate on top of those great APIs, or use tools like marvin. to help you automate some of the processes that are required to build an online data warehouse. In Part 2 and 3,  we will discuss the most valuable sections of a Data Warehouse, Analyzing and Visualizing your Data.

Analyzing the Data

marvin.’s super powers come from Google BigQuery API, an analytical database on the cloud. marvin. automates the process of inserting and storing data uploaded in the previous step. Together, Google BigQuery and marvin. are the analytical engine for your data warehouse. Your new Data Warehouse is limitless in terms of size and analytical computing power. Even better, there is nothing to code, no servers to manage, and no hard drives to setup. To learn more check out our blog:  Google BigQuery: 7 fascinating facts.

With marvin., you create datasets, make tables that host data, and preview that data in a table format. The data is no longer raw. Instead, your precious data is stored like in a huge spreadsheet that can be analyzed very quickly. Think billions of records in seconds. Think Big Data. Think cloud bi.

What if your business only has a few thousand records? No worries. marvin. will manage small data just as well. After all, a small business does not need to analyze a billion records of its data to make informed decisions and leverage insight to compete with other businesses.

Your new analytical engine also allows users to interact with simple SQL statements. If your team does not know SQL, it is also compatible with multiple visualization engines like Tableau and BIME, popular programs that present your data graphically. Visualizing the data is the last piece of the 4 steps of the simplified process.  We will explore data visualization in Part 3.   

Building an Online Data Warehouse Part 1

Google Cloud Platform is a great option for companies looking to build a Data Warehouse on the cloud, with plenty of APIs from which to build.  Currently, the number of APIs which you can choose from to connect your applications to the cloud is growing by the day.  As of now, you have two options: hire a developer to code and integrate on top of those great Google and Applications APIs, or use tools like marvin. to help you automate some of the processes that are required to build a data warehouse.

marvin. is a tool from Bimotics that allows you to set up an online data warehouse without the need to code. Google has created the capability to host the biggest data warehouse in history, Google BigQuery, and marvin. is the bridge you need to make that process as easy as possible.  The steps below will help you get started.

What does it take to set up a Data Warehouse on the cloud? A simplified process for cloud bi would contain the following steps: 

1. Gathering the Data

2. Storing the Data

3. Analyzing the Data

4. Visualizing the Data

Gathering the Data

This first step has historically been the hardest, but advances in on-premise technology and Cloud applications like QuickBooks and SalesForce now provide methods to extract the data through APIs. These are often referred to as connectors. Most of these apps allow users to extract data in formats like CSV or JSON. Even e-commerce platforms allow users to extract their customer, product, order, inventory, and lead information through these connectors. Extracting these files to build a Cloud Data Warehouse is where it all begins. Without access to raw data, it is hard to design and build the right Data Warehouse architecture. Remember the old mantra “garbage in, garbage out”, bad data will always yield bad insight or analysis. Important note before going any further: if your data has exceptions, missing items, or other errors, fix them in the application itself before progressing with your warehouse.

Storing the Data

Storage is another big challenge.  You will need to answer questions such as:

• Where should I store this data and how?
• Is there enough space on my servers and hard drives?
• How much data needs to be stored?
• How often does information need to be added?

After setting marvin. up, pull data from your chosen application (e.g. e-commerce sample). marvin. then allows you to create buckets or folders where these files are to be stored- each relating to specific customers, products, orders, inventory etc.

Organizing data into buckets makes it much easier to refer to later. We suggest you create two types of buckets for each file: one to process into the next step and another for files that have already been processed. For example, we have a file called “Orders_New” and one called “Orders_History.” Not to go into great detail on file names, but naming them as order_YYYYMMDD.csv can help identify when the orders where extracted.

marvin. allows you to store data in any format, but the next step requires that the data be either  in CSV or JSON formats. marvin. will take an additional helpful step in compressing the files into GZIP. This minimizes the storage space used as well as the cost associated with storing the data. Storing your data this way allows you to keep a good archive of your data. marvin. also lets you to upload multiple files or folders into a bucket. Finally, you can preview files, as well as download or delete the ones that you do not need.

In the next part of this series, we will go into the next two steps: analyzing and visualizing data using marvin. 

Bimotics 2014 in Review

2014 has been a productive year for Bimotics. While we have put in the long hours, as founders, we feel the work is never done.  Armed with optimism, we look forward to reaching new milestones in 2015.  2014 was full of challenges and successes and we have learned from all of them.   Reflecting on the year that was, here is the recap of our top 3 wins in 2014.

3. Implemented Marketing Automation: Set up flexible inbound and social marketing infrastructure  that the team could maintain and edit.  After evaluating several options, we chose  Hubspot as their inbound marketing and social media focus synergized well with our business model. With our increased marketing activities we were able to fully engage two campaigns which led to consistent growth in site traffic and lead generation.  We have really increased our visibility.  We are now receiving over 600 visitors a month to our website.  We are developing our marketing strategy to effectively hone in on even more channels to spread the word and gain further traction.

2. Became among the first Google Cloud Partners: Last year we became newly minted technical partners with Google Cloud Platform. We then got invited to their pilot certification program. In the first quarter, our technical founder Roberto completed the vigourous process of becoming among the first Google Cloud Certified Partners. With this new status, we have gained additional credibility within technical and investor circles.   More importantly, the relationship allowed us to successfully build a fully HIPAA compliant cloud environment for a client.

1. Launched our big data application,  marvin.:  Without having to code, our app gets your data warehouse on the world’s most powerful cloud in just minutes. marvin. is just the first step in realizing Bimotics’ vision of analytics for all businesses, but we are  on our way! Since launching in the fall we have attracted over 30,000 visitors and gained almost one user per day. We have learned so much about getting to market and we will continue to build on this early success.

With these accomplishments under our belt, we have the much needed stepping stones to keep Bimotics going into 2015. Business planning is well underway for next year and in our next segment we’ll outline some of our New Year’s Resolutions.  Until then, we wish you all a wonderful holiday season and a prosperous New Year.

Related Blogs:

marvin. now available in the Chrome Store

5 things we are looking for in our next employees