Catalog Completeness: a free PIM module for Magento Open Source

Thin product data, and no list of what to fix first#

A catalog with thousands of SKUs is never finished. Products arrive from supplier files with a SKU, a name and a price. Somebody pastes "See spec sheet" into the description. One image goes up, if any. Data entry works through the gaps alphabetically, or in whatever order the complaints arrive, because nothing ranks them.

The channels rank them for you, the hard way. Google Merchant Center's product data specification defines a required attribute in two sentences: "Submit this attribute. If you don't, your product won't be able to serve in ads and free listings." Meta's catalog fields reference says of its required fields: "If any of these attributes are missing or formatted incorrectly, products may not upload to your catalog".

Filled isn't finished. A nine-word description is filled. So is an 11-digit GTIN, and so is a weight of 0. A check that only asks whether a field is empty passes all three. Google's GTIN rules take 8, 12, 13 or 14 digits, and add: "Make sure that the check digit is present and correct."

Product completeness, scored#

Catalog Completeness is a free PIM module for Magento Open Source. It scores every product in your catalog from 0 to 100 against rules you write, weighs the rules that matter most, and turns the scores into a work list, lowest score first. It doesn't syndicate feeds to channels or hold product data outside Magento: the catalog stays in Magento, and the module measures it.

It's written for Magento Open Source 2.3 and 2.4. In the admin it sits under Big Cart Energy > PIM: Catalog Completeness, with three screens: Dashboard, Rule sets and Settings.

Weighted rules with data quality checks#

A rule set holds the rules for one or more attribute sets or, as a channel profile, for a sales channel. Each rule checks one attribute with one of six rule types, and only Present comes close to a filled-or-empty test:

Enter the pattern body only, without delimiters or modifiers: the editor refuses /.../i style input with that message. The pattern for that GTIN example saves as written and matches 8, 12, 13 and 14 digits:

^([0-9]{8}|[0-9]{12,14})$

A pattern checks the shape of a value, not its arithmetic, so it can't confirm a check digit. Keep a check-digit validator wherever GTINs enter your catalog.

Each rule carries a weight: Required counts 3, Important 2 and Nice to have 1. A rule's share of the score is its count divided by the total for the rule set, and the editor shows every share live as you weigh. Each rule also names the product types it applies to, all six by default, so a weight rule can skip virtual and downloadable products instead of failing them.

A product's score is the weight of the rules it passes, divided by the weight of the rules that apply to it, times 100, rounded. It reads 100 only when every applicable rule passes, so a product one rule short never rounds up to perfect. A rule on an attribute the product's attribute set doesn't have counts as failed, because the product can't hold a value. A product that no rule applies to has no score and stays out of the averages. The dashboard lists it under No rules assigned (below), and the product grid's Completeness column reads "No rule set" for any product without an attribute set score.

A worked example, from an invented catalog. A fasteners rule set has five rules: name present (Required), description at least 250 characters (Required), at least 3 images (Important), weight greater than 0 (Important, simple products only) and country of manufacture present (Nice to have). That's 11 points of weight. A hex cap screw, SKU HCS-M8-30-ZP, with a 60-character description and one image passes name, weight and country of manufacture: 6 of 11, a score of 55. The configurable product above it skips the weight rule, so it's scored out of 9.

Scores land in four bands, by default Complete at 100, Good from 80, Needs work from 50 and Poor below that, which puts the screw in Needs work. The Good and Needs work thresholds are settings.

The dashboard and the work list#

The dashboard answers three questions: how complete the catalog is, what's missing most, and which products to fix first.

• Five tiles: average score, products scored, Complete, Needs work or Poor, and the rule failing most. Each tile is also a filter.
• The distribution of products across the four bands.
• A breakdown by attribute set, category, brand or product type, lowest average first. Brand reads the attribute you name in the settings, manufacturer by default.
• The ten most-missing rules, each linked to the products that fail it.
• The work list: every scored product, lowest score first, with its missing rules, keyword search, filters, a column chooser and 20, 50, 100 or 200 rows a page. Hover over a row, or reach it with the keyboard, to see everything it's missing. Tick rows to recalculate or export only those.

Filters for attribute set, category, brand, product type and band apply to everything on the page, and "Enabled products only" is on by default. The Rule set select at the top switches the whole dashboard between your attribute set rules and any channel profile, and a status line says when the scores were calculated and whether they're out of date.

Export CSV hands the filtered work list, with every missing rule per product, to whoever does the data entry. Export selected does the same for the rows you tick.

Products with no rules assigned#

A ranked list has a blind spot: a product no rule covers has no score, so it never shows up in the ranking. The Rule coverage select, beside Rule set, switches the dashboard to those products. Under the attribute set rules, that's every product whose attribute set no enabled rule set covers, or whose product type none of that rule set's rules applies to. Under a channel profile, it's every product whose type none of the profile's rules applies to. The select shows the count, and a notice above the tiles repeats it with a link whenever there are any.

In that view the work list reads "No rules assigned" in the Score cell. Filters on the products themselves still apply, while band, the tiles and the Score and Last scored filters step aside, because they need a score. Export CSV and Export selected list the products with "No rules assigned" in the Band column, for whoever assigns attribute sets or writes the rules, and Recalculate rescores nothing and says why. Coverage is read from the rule sets as they stand, so the count is right before any recalculation.

Channel profiles#

A channel profile is a rule set for a channel instead of an attribute set, and it covers every product. Pick the channel and the editor shows a Suggested list of the fields that channel commonly requires or recommends, each with a one-line note and a suggested rule. Click Add and a row appears with the role, the rule type, the parameter and a suggested weight filled in, and the attribute left empty, because only you know whether your GTIN lives in gtin, upc or ean. Nothing is pre-set: no hint becomes a rule until you click.

Every product then carries one more score for each enabled profile, next to its attribute set score.

Hints cover Google Merchant Center, Microsoft Merchant Center, Meta (Facebook and Instagram Shops), Pinterest, TikTok Shop, Amazon, eBay and Walmart Marketplace. Each list was written from that channel's own documentation, and the package's docs/channels.md names the page and the date it was read. Channels change their specs; when a hint and the channel disagree, the channel wins.

Two things get no hints. Requirements that change by category belong in your own rules once you know where you list: eBay's listing guide says of item specifics, "The contents of this section depend on the item category", and Amazon sets its attribute requirements per product type in its product type definitions. Distributor feeds follow no common specification, so they get a Custom profile with rules you write.

The module never connects to a channel or sends a feed. It scores the data in Magento; your feed extension or connector still does the sending. When the feed itself is the problem, Merchant Center feeds are part of our ecommerce SEO work.

On the product page and in the product grid#

On the product edit page, a Catalog completeness panel shows the product's score and band, each missing rule with a link to the field that fixes it, and the rules it passes. A select switches between the attribute set rules and your channel profiles, and Recalculate this product rescores it on the spot.

In the rule set editor, the same panel scores any product against the rules on the page, unsaved edits included, so you see what a rule does before you save it.

The product grid gets a Completeness column with the band beside each score. It sorts, and it filters by range.

Who it's for#

Catalog and ecommerce managers at B2B and industrial distributors: thousands of SKUs, supplier data of every quality, attribute sets that differ by product line, and more products than anyone can review by hand. Those are the stores we build B2B ecommerce for. What a store sells matters less than its catalog: if the data needs cleaning up, or a new sales channel is coming, the work list shows where to start. Composer accepts Magento Open Source 2.3.0 to 2.4.9 on PHP 7.1 to 8.5, and it has passed our bench on 2.4.9, 2.4.8, 2.4.7, 2.4.6 and 2.3.7-p4; we list each version as it passes, and any version not listed here is untested.

It doesn't support Adobe Commerce, it doesn't score per store view, it has no GraphQL API, and it never shows a score to shoppers. When the cleanup is bigger than your team, catalog data at scale (attribute cleanup, SKU reconciliation, distributor feeds) is part of our ERP integration work.

How to set it up#

Install it on a staging copy first, with a backup you've restored. Composer brings in bigcartenergy/module-core with it: the small shared module that holds the Big Cart Energy menu and settings tab for every module of ours, and does nothing on its own. Then enable both modules, register them and recompile, following Adobe's steps for installing an extension:

bin/magento module:enable BigCartEnergy_Core BigCartEnergy_CatalogCompleteness --clear-static-content
bin/magento setup:upgrade
bin/magento setup:di:compile
bin/magento cache:clean

The module:enable line is optional, because setup:upgrade enables a newly installed module, and it does no harm. In production mode, also run bin/magento setup:static-content:deploy after the compile.

On 2.3.0 to 2.3.4, Magento itself ignores the __disableTmpl guard the module puts on admin-controlled labels, just as those releases ignore core's own guards; Magento honors it from 2.3.5. It's one more reason to upgrade a store still on those lines.

Then, in the admin:

1. Create a rule set. Under Big Cart Energy > PIM: Catalog Completeness > Rule sets, click Create a rule set, choose Attribute set rules and pick the attribute sets it covers. An attribute set already "determines the fields that are available during data entry" (Adobe), so it's the natural unit for what complete means. Each attribute set can sit in only one rule set of this kind, which gives every product one main score.
2. Add rules. Take them from the Suggested list (name, description, short description, images, manufacturer, weight, meta description and country of manufacture) or write your own. Watch the attribute picker: when some of the rule set's attribute sets lack an attribute, it says so, and their products will fail that rule.
3. Weigh them. Keep a hint's suggested weight or pick your own: Required, Important or Nice to have. The shares update as you go, and the footer says how much of the score your Required rules carry.
4. Preview, then save. Search for a product in the preview panel to see its score under the rules on the page. Save stores the rules and marks the scores out of date; Save and recalculate also queues a rescore of every product the rule set covers.
5. Check coverage. Switch Rule coverage to No rules assigned. Every product it lists needs its attribute set in a rule set, or a rule for its product type, before it gets a score.
6. Work the list from the top. Open Big Cart Energy > PIM: Catalog Completeness > Dashboard, filter to a category or a brand, and start with the lowest scores, or export the list for whoever does the data entry.

Read the user manual for the long version: every screen, setting and admin notice, step by step, with screenshots.

Keeping scores current#

Three things rescore products: the indexer, when Magento saves them; the scheduled rescore, for changes that never pass through a save; and Recalculate, when you ask.

The indexer#

Scores live in a Magento indexer, Catalog completeness score (bce_completeness_score), so they follow the mode you set under System > Tools > Index Management, like any other indexer. In the words of Adobe's Index management page, Update on Save means "The index is set to update whenever an entity change is saved", and Update by Schedule means "The index is set to update on schedule according to a cron job".

• Update on Save (realtime): a product save, a mass attribute update or an import rescores the products it touches, in the same request.
• Update by Schedule (schedule): switching to it creates the changelog table bce_completeness_score_cl and adds the module's statements to the catalog tables' triggers. Magento's indexer cron then applies the changes, so cron has to run.

Which mode the indexer starts in depends on your Magento version: 2.4.8 and 2.4.9 put a new indexer in Update by Schedule, while 2.4.7 and earlier leave it in Update on Save, whatever the other indexers use. Check it after the install, then set the one you want with Adobe's indexer commands:

bin/magento indexer:show-mode bce_completeness_score
bin/magento indexer:set-mode schedule bce_completeness_score

For Update on Save, use realtime in place of schedule.

When an ERP sync skips the save#

Here's the catch. Product saves, mass attribute updates and imports rescore the products they touch. An ERP sync, a script or a migration that writes straight to the database is none of the three, so in Update on Save its changes reach the scores only when something rescores them. Update by Schedule's triggers do follow direct writes to the product and attribute value tables, but not the stock status or any table they don't watch.

The scheduled rescore closes that gap. It runs from Magento cron as the job bce_completeness_rescore, in its own cron group, bce_completeness, which runs in a separate process like Magento's indexer group. It's on by default, every six hours. Its settings are under Stores > Configuration > Big Cart Energy > PIM: Catalog Completeness, in the Periodic rescore group:

Pick Products changed since the last run only if your sync sets updated_at. Magento's own saves and imports set it, but many direct database writes don't, and a product they change keeps its old score until a full pass rescores it.

The group needs Magento cron to run. Adobe's cron:run runs every group when you leave out --group ("omit this option to run cron for all groups", from Adobe's cron guide), so a crontab that runs plain bin/magento cron:run picks the new group up, and one that names its groups needs a line for bce_completeness. To run the group once by hand:

bin/magento cron:run --group=bce_completeness

Each run writes one line to var/log/bce_completeness.log: what it covered, the products scored, the batches and how long it took. A run that fails logs how many products it scored before the error, and Magento marks the cron job as failed. The start of the last completed run is kept in Magento's flag table under bce_completeness_rescore, and the changed-since mode counts from there.

Recalculate on demand#

The dashboard's Recalculate and the editor's Save and recalculate queue the work in Magento's message queue, on the bce_completeness.recalculate topic, and the dashboard counts the products done while it runs. A queued job waits for a consumer, though, and Adobe's message queue documentation says "Cron jobs are the default mechanism to restart consumers." No cron, no rescore, and the dashboard says so when no consumer has picked a run up. Magento's consumers_runner cron job starts the consumer, bceCompletenessRecalculate. To start it yourself:

bin/magento queue:consumers:start bceCompletenessRecalculate --single-thread --max-messages=10

If cron isn't set up, or cron_run has switched the consumers_runner job off, recalculate from the command line instead. It runs in the foreground with a progress bar:

bin/magento bce:completeness:recalculate
bin/magento bce:completeness:recalculate --ruleset=3
bin/magento bce:completeness:recalculate --product=1042

The first line rescores every product; the other two take one rule set or one product, by id.

How it fits into your store#

The six tables are bce_completeness_ruleset, bce_completeness_ruleset_attribute_set, bce_completeness_rule, bce_completeness_score, bce_completeness_failure and bce_completeness_run. The changelog table bce_completeness_score_cl exists only while the indexer runs in Update by Schedule.

In User Roles, the rights sit under Big Cart Energy > PIM: Catalog Completeness as Dashboard, Rule sets, Edit rule sets, Recalculate scores and Export CSV. The settings right is PIM: Catalog Completeness Section, under Stores > Settings > Configuration.

The routes are GET /V1/bce-completeness/products/:productId/scores, GET /V1/bce-completeness/rulesets and POST /V1/bce-completeness/products/:productId/recalculate; the POST rescores that one product at once and returns its scores.

Every score entry carries a state: scored, no_rules_assigned when no rule applies to the product, or not_calculated when rules apply and no score exists yet. Read state, not score: Magento's REST output leaves out null values, so an entry without a score has no score, band, band_label or scored_at key at all. When a product has no score from the attribute set rules, the scores and recalculate routes start with one entry for them, with is_primary true, ruleset_id 0 and the state that says why, followed by the channel profiles' entries.

Products loaded through the product repository, in the admin and over REST, carry three read-only extension attributes: bce_completeness_score (the attribute set score, left out when there is none), bce_completeness_missing (the labels of the rules it fails) and bce_completeness_state (the same three states, for that score). Sending them on a product save does nothing.

The settings live under Stores > Configuration > Big Cart Energy > PIM: Catalog Completeness, which Big Cart Energy > PIM: Catalog Completeness > Settings also opens, in three groups: General (the brand attribute, whether the dashboard starts on enabled products only, the work list's rows per page and the CSV row cap), Score bands (Good from and Needs work from) and Periodic rescore (the four settings above).

The user manual holds the full reference: the REST responses in full, every admin notice with what to do about it, and a quick reference of paths and commands.

How to remove it#

Same rules as the install: a staging copy first, with a backup. There are two ways out, and on the second, the order of the commands decides whether your data survives.

Remove the module and its data#

Run Magento's uninstall command:

bin/magento module:uninstall BigCartEnergy_CatalogCompleteness --remove-data

It runs the module's uninstall routine. That first takes the indexer out of Update by Schedule, so the catalog tables' triggers stop naming the module and the changelog table goes with them. Then it drops the six tables, removes the indexer, queue, configuration, cron schedule and flag rows, and removes the package with Composer. Adobe documents the command under Uninstall modules, including that it puts the store in maintenance mode while it runs. Product saves keep working during and after the uninstall, in either indexer mode, and the log file var/log/bce_completeness.log stays until you delete it.

module:uninstall works for packages installed with Composer, which this one is, and it uninstalls this module only. BigCartEnergy_Core stays installed and enabled while something else requires bigcartenergy/module-core: another Big Cart Energy module, or your store's own composer.json. When Catalog Completeness was installed alone, as the Install section above does it, Composer removes module-core with it as an orphaned dependency; the --no-update-with-dependencies flag Magento passes to Composer doesn't keep it. To keep the core module, run composer require bigcartenergy/module-core before the uninstall. In production mode, follow the uninstall with bin/magento setup:di:compile and bin/magento cache:clean, as after any module change.

Remove the code and keep the data#

Disable the module, remove the package, then run setup:upgrade, in that order. The six tables and the module's settings stay in place:

bin/magento module:disable BigCartEnergy_CatalogCompleteness
composer remove bigcartenergy/module-catalog-completeness
bin/magento setup:upgrade
bin/magento setup:di:compile   # production mode
bin/magento cache:clean

On this path, with no other Big Cart Energy module installed, Composer removes bigcartenergy/module-core too, as a dependency nothing needs, and setup:upgrade drops Core from Magento's module list. To say so explicitly, name both: bin/magento module:disable BigCartEnergy_CatalogCompleteness BigCartEnergy_Core, then composer remove bigcartenergy/module-catalog-completeness bigcartenergy/module-core. If another module depends on one you disable, Magento stops and shows the dependency in its error (Adobe).

Installation and configuration service#

The module is the whole product. Every feature on this page is in the free package, open source under OSL-3.0, and none of it waits behind a paid version. What we sell is the setup, and the judgment the software can't make for you:

• Installation, on your staging copy first, then production.
• Rule sets designed for your attribute sets and for the channels you sell on.
• The weights and quality rules: which fields are Required, how long a description has to be, and what a valid value looks like in your data.
• A cleanup plan from your first scored run: what to fix first, and in what order.
• A hand-off to your catalog team, so they can run it without us.

Concretely, we write the rule sets your attribute sets need and a channel profile for each channel you feed, point every role at the attribute that really holds it, set the rescore's schedule and mode around how your ERP or supplier syncs write, and turn the first scored run into the cleanup plan: lowest scores first, plus every product with no rules assigned.

Like every engagement, it's scoped in writing for your catalog, and we don't publish prices; how an engagement runs covers the rest. Send your Magento version, a rough count of SKUs and attribute sets, and the channels you feed through the form below. We reply with a plan, not a pitch.