I went into this wanting to create a step-by-step guide for effective keyword research. In researching this piece, I read 89 great, great guides and expert articles about keyword research post RankBrain. I've included tips and advice from the best influencers in SEO and only selected methods that lead to quick results.
In this step-by-step guide, you will find the basics, as well as advanced techniques recommended by some of the world's leading SEO experts, including:
The experts featured here represent a total of 1.2 million followers on Twitter!
Alright, let's get started!
Always start with numbers
In this SMX SlideShare, Casie Gilette provides the statistics you should have in mind before even starting your keyword research:
- 50% of search queries are 4 words or longer
- 27% of search queries are questions
- Average voice search query is 5-6 words
Keyword research has changed a lot over the years. Here are the recommended steps, post RankBrain.
Keyword Research Basics
The notion of keywords
In this guide, when I use the term "keyword," it can mean a single word or several. For example, "camp" might be a keyword but "cheap summer camp" can also be a keyword.
Before starting your keyword research, it's important to structure your keyword list correctly. A keyword list isn't just a long list of all the keywords related to your activity. It has to be organized so that you can start working with it right away. If your goal is to get quick results, then you have to organize your keyword research very efficiently. Your first keyword list should be structured something like this:
Let's explore each column of this keyword list.
"Topic" can mean different things, depending on the type of website you're optimizing.
- a category of product, e.g. "ultraportable laptops"
- a subcategory of product, e.g. "ultraportable laptops with ssd"
- a specific product that a many people are searching for, e.g. "Dell XPS 13"
- a blog post, e.g. best ultraportables
- a question, e.g. "what ultraportable should I buy"
- a theme, e.g. "Spring cocktails ideas"
- a problem, e.g. "broken iPhone"
- a specific usage of your service. For example, the same place could rank for "wedding venue", "seminar venue", "meeting space," and "conference room."
- a subsection of the services you offer: "horseback ridding", "riding lessons", "horse riding club"
- an activity in a specific place (general topic + location), e.g. "meeting space new york city"
- different activities in different locations (specific topics + location), e.g. "horse riding club new york," "horse riding club chicago," "horseback ridding new york", "horseback ridding chicago," etc.
Search volume: use it to rank your keywords, not to predict future traffic
Most keyword tools usually provide a metric called search volume, which comes from Google AdWords' API. However, search volume is not always accurate, which Russ Jones explains very well in this excellent article. But right now, we're more interested in knowing how to prioritize different keywords than predicting precisely how many visitors each query might yield.
In any case, when you target a specific keyword, only 10 to 20% of your traffic to that page/topic will be a direct result of that targeted keyword; the rest of your traffic will come from different combinations of this keyword and long-tail expression containing the keyword or its synonyms.
For each topic, you will want to:
- Find a list of derivative requests users might type into Google.
- Rank those keywords by search volume
- The keyword with the highest volume will be the primary keyword for this topic
- The remaining keywords will be secondary keywords.
I will delve into this hierarchy a little bit deeper later on.
For each topic, you will need to create a specific landing page. In this column, enter the URL of the page you created to target that specific topic.
Step 1: Brainstorm
SEO influencers recommend brainstorming first (see this infographic from Search Engine Land). The goal here is to come up with your first keyword ideas that will develop and evolve in the following steps using keyword research tools.
I'd like to add a personal piece of advice here. I've created five companies, and each time, I was surprised at the way my clients defined what I do. So I have one universal rule: entrepreneurs don't know what words clients will use to find the businesses they run. This is why the main task of any entrepreneur is to meet their clients and listen to them. SEO should always start with asking clients how they would define your product/activity and carefully noting the words they use.
At Moz, they used Google Custom Audience Surveys to ask customers which keyword they were using.
Eric Enge's advice is to use your internal site search for inspiration. If you can get a list of keywords and phrases that your existing customers are already using to find what they need on your website, you'll have an easier time creating your first keyword list.
Having finished the first step, you should now have a list of 10-100 keywords on your spreadsheet.
Step 2: Add keywords that people already use to find existing pages of your website
Bruce Clay's advice is to start with what you already have. This is good advice because it's always going to be faster to optimize and improve existing results then to create new goals. To do so, start by studying the insights from your Google search console:
- Set up your search console
- Log into your search console
- Go to Search Traffic > Search Analytics
- Note all the keywords that interest you on this list.
Some influencers also suggest listing your existing landing pages and finding the keywords that are specific to those landing pages. I personally don't think that's a good idea because the goal of keyword research is to define a new ideal structure for your website, not just polishing up the status quo.
Marie Haynes says that she likes using the Google search console to find keywords that bring traffic even when the website isn't currently ranking in the top 3. It means that those keywords have high potential if you focus on them.
You should now have a complete list of new keyword ideas.
Step 3: Brainstorm with competitors' keywords
Select some competitors that rank well for the first keywords you found. To find these, just type those keywords in a private browser tab (Incognito on Chrome). Make sure you've logged out of all your Google accounts and cleared your cookies and cache.
Then, you can use a competition-tracking tool to find the keywords your competitor uses.
Below are a few competition-tracking tools you may want to check out:
Mazen provides a list of keywords that the competitor currently ranks for, as well as a list of keywords that the competitor should target based on the semantics of their website (homepage, landing pages, etc.).
SEMrush is more about estimating a competitor's existing traffic (which is sometimes accurate) and providing a list of keywords that bring this traffic. For each keyword, you get the search volume, the landing page that is currently ranking, and the number of Google results for each keyword (which helps to estimate the difficulty). SEMrush will also help you discover other competitors by displaying your competitors' competitors.
With SpyFu, you can find an exact list of your competitors' keywords, as well as each keyword's ranking and pertinent data about each keyword. SpyFu also provide an estimated number of visits (clicks) that each keyword yields for the competitor.
At this point, you should have more keywords in your list, as well as relevant data for each keyword (difficulty, cost per click, and search volume), which can be obtained using dedicated tools.
Step 4: Find new keyword ideas with dedicated tools
You can literally find hundreds of different keyword suggestion tools online. I even found this list of 117 keyword tools compiled by Brian Lang.
Here are the features you should look for in these tools:
In Mazen and SEMrush (under "Keyword Magic Tool"), the keywords suggested are pre-grouped into categories. For example, if you type "web agency", you will get categories like design, marketing, SEO, digital marketing, web development, etc.
Here's a step-by-step guide to grouping keywords in Google AdWords:
- Log into your Google AdWords account
- Click on "Tools" > "Keyword Planner",
- Click on "Search for new keywords using a phrase, website, or category"
- Enter your keyword idea
- Select the "Ad group ideas" tab
Artificial intelligence keyword suggestions:
When you enter a keyword to get suggestions into most keyword research tools, you'll mainly get variations of the same keyword (see below):
In these examples, each keyword suggested contains at least 1 of the 2 words that compose the original keyword idea "web agency" and often both. It's more about completing those keywords than finding new ideas.
That's why you should use tools that not only provide variations, but AI recommendations as well.
Mazen produces suggestions that don't have any common words with the original. For example, if you type "web agency," you might get suggestions like "web design company," "digital marketing company," "marketing website design," "ecommerce website designer," etc.
But the really interesting feature is that it will also generate keywords for the entire business scope of a web agency: web design, digital agency, web marketing, SEO services, small business design, affordable web design, e-commerce, etc.
SEMrush also provides strong results. In the screenshots below, you can see that the tool has found some new specific topics like wordpress agency, web development, web marketing, e-commerce, etc.
Step 5: Add questions
Google's RankBrain algorithm, along with the rise of vocal search, has turned Google into a lean mean answering machine. Structuring your blog according to questions that your clients/users/readers are asking is a very good way to find new content ideas and get more qualified traffic.
Matt Janaway's advice, and I totally agree with him, is to use AnswerThePublic to get ideas for the questions you'd be answering. BuzzSumo is another tool that allows you to find questions people might ask. BuzzSumo is very helpful because it shows you topics that have trended and received social shares or backlinks.
Here's an example from AnswerThePublic:
Here, we can see that there is an opportunity with questions like "what do summer camps cost?" or "what are some good summer camps?" We would then write some great content that answers those questions and includes comparisons, infographics and/or videos.
Here's an example using BuzzSumo :
This tells us that "adult summer camp" is a great idea for content because it generates a lot of social sharing.
Step 6: Group keywords into primary and secondary
The idea of primary and secondary keywords is probably the most important notion in keyword research. A primary keyword is a general topic, whereas a secondary keyword is a variation of a primary keyword. It can also be a completely different phrase with a similar meaning.
According to a study on RankBrain by Ahrefs, it is now possible to rank for hundreds or thousands of keywords with only one page (i.e. one topic) if those keywords all target exactly the same topic. That means you don't have to use every single keyword in your content because Google now understands that close semantic keywords have the same meaning. Some aspects of keywords that Google might treat the same include:
- Different word order: "apartment rentals New York" and "New York rental apartments"
- Singular/Plural: "rent a flat in new york" and "rent flats in new york"
- Abbreviations: "rent a flat in New York" and "rent a flat NYC"
- Synonyms: "cheap flat NYC" and "affordable flat NYC"
In a fantastic article published on Moz's blog, Dr. Peter J. Meyers explains how RankBrain is changing how keyword research is performed. What are the consequences for this new keyword research paradigm?
- You don't need to do cram keywords into your text (i.e. keyword-stuffing): Google understands the meaning of your content, so you should only use variations/synonyms, but don't obsess over it.
- You need to create content that helps Google understand your purpose: this means longer content, but also semantically relevant content. You can read more about that here.
- Your page should rank not only for one main version of your key phrase but also for dozens of variations. The advice here is the same that SEO specialists had for Google's Panda algorithm. Yulia Khansvyarova, head of SEMrush Marketing, says "Stop creating pages or content tailored to only one keyword or keyword phrase". That is to say, target general topics that will rank for a wider range of keywords. I would add that, with RankBrain, it's essential to track your ranking for a large number of variations of your main phrase.
Take a look at what I mean below:
In this example, the first keyword is "apartments for rent nyc" and it has the highest search volume. If we only considered the search volume for this topic, we would be missing a lot because there are tons of variations of this general topic. The global search volume of the entire topic here is 61,180 (and probably much more, since only a few examples of keywords are displayed here), which is a lot more than the search volume of the primary keyword (33,000). This means that you need to add up the search volume of each variant in order to obtain the real search volume for this topic (or a closer estimate).
The second reason these secondary keywords are so important is that you need to track your ranking not only for the primary, but also for each secondary. If you only rank for a small segment of the keywords, it probably means that your page is not perfectly relevant and that you should consider improving your content. If you rank for the primary but also for 70 to 80% of the secondary keywords, then your final traffic for the page will be far more significant.
When researching keywords, you should really create a hierarchy within your list:
There are two keyword tools that allow you to easily group your keywords into primary and secondary buckets:
Mazen provides different variations for each keyword. You don't need to group the keywords yourself because Mazen's algorithm does it for you. Keywords are then sorted into primary and secondary groups.
The advantage here is that Mazen automatically detects the different topics you should target, and the different keywords (variations) that you should use for each.
Take a look at this example:
Moz also has a feature that allows you to group keywords. To use it, go to keyword explorer, and, in the "Group Keywords" dropdown menu, select "yes, with high lexical similarity":
Step 7: Organize keywords
Keyword organization is probably the most time consuming task you'll encounter in SEO. But it's important to do it properly, because if you don't:
- You won't be able to prioritize your work
- You won't be able to track your rankings and make decisions based on them
- You won't be able to optimize your content properly and evaluate the results of those improvements
The minimum information you need
Each SEO specialist has their own way of organizing keywords and adding information about them. But all the big SEO minds agree that you should, at the very least, include the following:
Let's explore what each data point means and how to find it.
We already explained search volume and why you need it at the beginning of this guide.
Keyword difficulty is important because it allows you to prioritize your work. An SEO campaign is very time consuming. It's not really worth it to spend days trying to rank on a very difficult keyword if you can easily bring new customers to your site with easier keywords. You'll especially want to prioritize easier queries when your website is young and you don't have a lot of backlinks to your domain.
There are three main ways to calculate keyword difficulty
The first method is to rely on Google AdWords difficulty. This method is not very relevant for SEO because AdWords only calculates difficulty based on the number of companies that buy that keyword on Google AdWords. But interestingly enough, the most difficult AdWords keywords are sometimes quite easy to target via organic SEO.
The second method of evaluating keyword difficult is based on the total number of internet pages that have been built to specifically target a keyword. In this case, you would measure the number of pages that have made SEO efforts to target that keyword. To find that number, use and combine any of the following three indicators:
- Number of search results for this keyword
- Number of search results that have the keyword in the title tag
- Number of search results that have the keyword in the URL
This method can help you figure out how many competitors are targeting the keyword, but not how serious any of those competitors might be. For example, it could end up being more difficult to rank in the top 10 for a keyword targeted by 15 competitors who have a lot of backlinks to their pages, than to rank in the top 10 against 1,500 competitors that don't have many backlinks.
Tim Soulo, head of marketing at Ahrefs, says that, although this method worked in 2010, it isn't as effective anymore. He explains that with Hummingbird and RankBrain, the exact keyword you use in the title tag isn't that important since Google can understand the general meaning of your content even if the exact term isn't found on a page. Tim Soulo explains it like this:
Check out the SERP for the keyword 'guest writing' to see what I'm talking about:
Clearly, Google understands that things like 'guest writing,' 'guest blogging,' and 'guest posting' are closely related.
The last (and I think best) method of measuring keyword difficulty is to tally the backlinks and page authority of the top 10 Google search results for that keyword. This method is used by Ahrefs and SEMrush, and I strongly recommend leaning on these metrics. The more backlinks and authority pages in the top 10 have, the more difficult it will be for you to rank on that keyword.
To fill out this column, you'll need to answer two questions:
- Is there already a page ranking for the primary keyword?
If yes: you can optimize the content to rank better
If not, you should ask a new question:
- If there are no pages ranking for the primary keyword, is there a page on your website dedicated to that very topic?
If there isn't, now your know you need to create a new page to target that topic.
If your website is huge, this process can take a very long time. And if it's not your website, it can be difficult to answer the second question. The good news is that Mazen has an algorithm integrated in its software that automatically finds the best landing page for each primary keyword… which can save you a lot of time!
To prioritize your work, you'll need to know which keywords are already ranking, and at which position. According to LinksManagement, keyword traffic distribution looks like this:
This means that a keyword that ranks 5th or 6th would get twice as much traffic if it was ranked 3rd, and three times as much if it was ranked 2nd. If you make improvements on the pages ranked in the "golden triangle" (the top 3), you can get some quick wins that will help you save a lot of time early in your campaign.
Save 90% of the time spent on organizing keywords
To get any SEO campaign off the ground, you will need to gather information from various sources, including:
- Keyword research tools for keyword lists and volume/competition research
- Keyword tracking tools for rankings
- Google analytics for traffic
- SEO tools for backlinks
After that, you'll also need to organize your keywords into groups of primary/secondary keywords and topic categories.
These operations are very time consuming. That's why you should use a comprehensive tool like Mazen to organize your keywords, pages, and data without having to export a single Excel file. In Mazen, you can group and categorize keywords with a single interface.
The software automatically finds the pages that rank (or should rank) for each primary keyword.
Other columns you might want in your spreadsheet
SEO is a craft: we each have our own techniques and priorities. Here are some of the data columns most of our experts use:
If you can get this information (specifically for e-commerce), then you'll be able to prioritize the job on keywords/pages that are converting the best. This helps a lot, because it's probably not worth the effort to target keywords that don't convert. But you should also think about other types of "conversions" and goals… For example, a keyword that brings in a lot of traffic but few conversions can still be useful if it attracts backlinks.
Type of snippets available on the SERP
Depending on the keyword, there might be different snippets available in the search results. Larry Kim says that click-through rate (CTR) on the SERP is one of the most important ranking factors Google uses with RankBrain. He adds that the best way to generate traffic is to have a great CTR. It's very useful to know how the SERP is structured for each keyword:
- Is there an answer box ? ("Position 0")
- Is there a video or image snippet?
- Is there a map, Google shopping, or customer review snippet ?
Knowing what the SERP looks like will help you create the best rich snippet for your keyword's landing page, as well as use the right media for that page.
Some SEO influencers will advise you to add a column where you can rate a keyword's business potential. This is sometimes called "user intent." This can be useful in prioritizing your work. For example, let's say you have a hotel booking website; a customer searching for "what is the best destination in California?" is not as likely to convert as a customer searching for "affordable hotel San Francisco".
Brian Dean says that commercial intent is one of the more important elements of keyword research. He explains that one of the first websites he created generated 60,000 unique visitors per month, but only $400 per month in revenue. The reason was that, when he researched his keywords, he only focused on search volume and not on at all on commercial intent.
To my mind, this method should also be complemented with "backlink potential." Some keywords might lead you to create very interesting and high value content that is more likely to interest other bloggers/websites than generate revenue. This content can become good link baiting material.
As you probably know, internal linking is one of the most important elements of on-page optimization. But it's really important to plan this out thoroughly. If you want to use internal linking effectively, check out this piece I recently wrote about the subject.
The general idea is that you need to group your pages into semantic clusters. This means that your primary keywords (topics) should have a logical hierarchy. When it comes to on-page optimization, it will help you figure out which pages to link to one another.
Advanced techniques: The best of what's in the crazy minds of the greatest SEO influencers
Branded keywords of your competitor
Why not target your competitors' brand keywords? For example, if you were competing with Evernote, you could target a keyword like: "how to use Evernote". To find such keywords, you can use Google's related searches.
Persona-driven keyword research
Creating your SEO keyword list is a process not unlike market research. Michael King explains how iPullRank totally re-built its keyword research around business personas. The step-by-step guide for this method is:
- Create personas for different types of prospective Googlers
- Create surveys for your customers to fill out. King explains: "We develop questions that allow users to self-identify or even disqualify themselves as our target personas." They then flesh out these user profiles by asking respondents about their search behavior in order to figure out what keywords they might use and where they are in the decision-making process.
- Extrapolate the data. iPullRank takes the data they collected from the survey and applies that logic across all of the keywords in their list