# are there non-portable npm modules? or can something be done about them?

**URL:** <https://talk.observablehq.com/t/are-there-non-portable-npm-modules-or-can-something-be-done-about-them/2593>\
**Category:** Help\
**Created:** [November 29, 2019, 11:28pm UTC](https://talk.observablehq.com/t/are-there-non-portable-npm-modules-or-can-something-be-done-about-them/2593 "2019-11-29T23:28:03Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![aaronkyle](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/aaronkyle/32/8106_2.png) [@aaronkyle](https://talk.observablehq.com/u/aaronkyle)\
**Post date:** [November 29, 2019, 11:28pm UTC](https://talk.observablehq.com/t/are-there-non-portable-npm-modules-or-can-something-be-done-about-them/2593/1 "2019-11-29T23:28:03Z")

</div>

I am trying to learn web scraping using [Apify SDK](https://sdk.apify.com/). The [basic example](https://sdk.apify.com/docs/examples/basiccrawler) for this utility loads in two modules, then executes a short scraping script:  
module 1) [apify](https://www.npmjs.com/package/apify)  
module 2) [request-promise](https://www.npmjs.com/package/request-promise)

[The module require debugger](https://observablehq.com/@tmcw/module-require-debugger) succeeds in loading these modules via the Global Leaks Pattern:

`Apify = require('apify@0.17.0/build/index.js').catch(() => window["_events"])`

`requestPromise = require('request-promise@4.2.5/lib/rp.js').catch(() => window["Bluebird"])`

However, when plugging in the basic example scraping script, it fails with an error `TypeError: Cannot read property 'main' of undefined`.

The [console](https://developers.google.com/web/tools/chrome-devtools/console) identifies that the apify module attempts to load subsequent ‘require’ operations, but that it fails—presumably b/c they are written to use node + npm and need alternative `require` patterns in Observable?

To be specific, one of two places the console is telling me that it is failing is when loading this index.js (apparently from within the apify module loaded via npm):

```auto
"use strict";

var _events = _interopRequireDefault(require("events"));
var _log = _interopRequireDefault(require("apify-shared/log"));
var _consts = require("apify-shared/consts");
var _actor = require("./actor");
var _autoscaled_pool = _interopRequireDefault(require("./autoscaling/autoscaled_pool"));
var _basic_crawler = _interopRequireDefault(require("./crawlers/basic_crawler"));
var _cheerio_crawler = _interopRequireDefault(require("./crawlers/cheerio_crawler"));
var _dataset = require("./dataset");
var _events2 = _interopRequireWildcard(require("./events"));
var _key_value_store = require("./key_value_store");
var _puppeteer = require("./puppeteer");
var _puppeteer_crawler = _interopRequireDefault(require("./crawlers/puppeteer_crawler"));
var _puppeteer_pool = _interopRequireDefault(require("./puppeteer_pool"));
var _request = _interopRequireDefault(require("./request"));
var _request_list = require("./request_list");
var _request_queue = require("./request_queue");
var _settings_rotator = _interopRequireDefault(require("./settings_rotator"));
var _utils = require("./utils");
var _puppeteer_utils = require("./puppeteer_utils");
var _utils_social = require("./utils_social");
var _enqueue_links = require("./enqueue_links/enqueue_links");
var _pseudo_url = _interopRequireDefault(require("./pseudo_url"));
var _live_view_server = _interopRequireDefault(require("./live_view/live_view_server"));
var _utils_request = require("./utils_request");
var _session_pool = require("./session_pool/session_pool");
var _session = require("./session_pool/session");
... [I cut the rest for space considerations]

```

**So what to do?** _Is it possible to get around this by manually requiring in each of the failed modules within modules? Or is this not advisable?_

I’d greatly appreciate any insights.

Here’s my attempt at reproducing the example:

> **[Apify SDK](https://observablehq.com/d/a1cb8df20eed8c4f)**
>
> Apify SDK = The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

Thank you!

**Other References Consulted:**

> [@Module tips and tricks](https://talk.observablehq.com/t/module-tips-and-tricks/450/):
>
> Hi all, Are there any tips and tricks for non-UMD modules - in particular, any handy utilities to get the packages to play nicely with Observable? Here are a couple of examples: [https://github.com/keithwhor/NtSeq/blob/master/web/ntseq.js](https://github.com/keithwhor/NtSeq/blob/master/web/ntseq.js)[https://github.com/mattbierbaum/zombies-usa/blob/master/js/binaryheap.js](https://github.com/mattbierbaum/zombies-usa/blob/master/js/binaryheap.js) Best, Simon

> **[How to require stubborn modules | Observable documentation](https://observablehq.com/documentation/debugging/require-stubborn-modules)**
>
> All your Observable questions answered

> **[JavaScript cell | Observable documentation](https://observablehq.com/@observablehq/observables-not-javascript)**
>
> All your Observable questions answered

[require.js : WHY AMD?](https://requirejs.org/docs/whyamd.html#commonjs)

---

<div class="post-metadata">

**Author:** ![aaronkyle](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/aaronkyle/32/8106_2.png) [@aaronkyle](https://talk.observablehq.com/u/aaronkyle)\
**Post date:** [December 1, 2019, 2:07pm UTC](https://talk.observablehq.com/t/are-there-non-portable-npm-modules-or-can-something-be-done-about-them/2593/2 "2019-12-01T14:07:10Z")

</div>

Not that this has anything to do with my specific question, but if ever in the future anyone is looking into web scraping on Observable, here’s an easy-to-use point of departure:

> **[In-browser data scraping](https://observablehq.com/@bsaf/in-browser-data-scraping)**
>
> A quick example of how we might grab and process some data. First, start with a URL you want to scrape from: You might need to add CORS anywhere to the start of it: https://cors-anywhere.herokuapp.com/ We now get the HTML of the page using Axios: And...

In 2018, there was a [case brought against IFC in the US Supreme Court that resulted in its loss of certain (specific) immunities](https://www.supremecourt.gov/opinions/18pdf/17-1011_mkhn.pdf). ADB co-financed this project with IFC. For fun, I did a quick scrape of the associated project data on ADB’s website (roughly following Basil’s approach):

> **[Data-Scraping ADB's Project Page](https://observablehq.com/d/e6d6b9d87932091f)**
>
> Appendix

One thing I learned in this is that ADB’s project data pages aren’t all the same, and these scraping tools break pretty easily. ☹ oh well. In case it’s helpful to anything, I’ll share it here (unlikely that I’ll ever ‘publish’ it in my feed, as it’s pretty case-specific).

I am saddened at how difficult it is to cull data from the web. For all this work, sometimes I feel I was working faster & more efficiently by just copying and pasting with my mouse. Grr.

Anyway - if anyone can help with my module woes - that’d be awesome!! 🙏

---

<div class="post-metadata">

**Author:** ![bgchen](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/bgchen/32/784_2.png) [@bgchen](https://talk.observablehq.com/u/bgchen)\
**Post date:** [December 5, 2019, 11:57pm UTC](https://talk.observablehq.com/t/are-there-non-portable-npm-modules-or-can-something-be-done-about-them/2593/3 "2019-12-05T23:57:25Z")

</div>

Hi Aaron, I’ve been hoping someone with more module / Apify experience would answer, but since no one has, I guess I will make an attempt:

> [@aaronkyle](#):
>
> The [console](https://developers.google.com/web/tools/chrome-devtools/console) identifies that the apify module attempts to load subsequent ‘require’ operations, but that it fails—presumably b/c they are written to use node + npm and need alternative `require` patterns in Observable?

Yes, that’s correct. Apify can only be used with node on your computer or a server; there’s no way to run it in the browser. While some node packages can be bundled to run in the browser using tools like browserify, webpack and rollup, this is not feasible for Apify, which has to be able to launch and control instances of headless Chrome.

---

<div class="post-metadata">

**Author:** ![aaronkyle](https://yyz2.discourse-cdn.com/flex030/user_avatar/talk.observablehq.com/aaronkyle/32/8106_2.png) [@aaronkyle](https://talk.observablehq.com/u/aaronkyle)\
**Post date:** [December 6, 2019, 12:40am UTC](https://talk.observablehq.com/t/are-there-non-portable-npm-modules-or-can-something-be-done-about-them/2593/4 "2019-12-06T00:40:41Z")

</div>

Thank you, Bryan. I had a feeling this was the case, but haven’t gotten as far as I would like into understanding modules just yet to fully confirm. During this experience, however, I have enjoyed reading about [CommonJS](http://www.commonjs.org/) and [AMD](https://github.com/amdjs/amdjs-api/wiki/AMD) and the related [require documentation](https://requirejs.org/docs/commonjs.html). I haven’t quite figured out how to tell which is which when reading module information on npm, so I was sorta of wondering if that was part of the issue. I also tried requiring in the many modules listed in that index file independently, but that turned out to be a somewhat absurd exercise…

I also owe @mhebrard a follow-up in the thread [save a map (Openlayers)](https://talk.observablehq.com/t/save-a-map-openlayers/2598), where he identifies the module [`html-to-image`](https://www.npmjs.com/package/html-to-image) as not loading correctly… and I have been suspecting it’s something of the same issue (?)…

Thank you for your time and for helping me to stop messing around too much on this one! 🙂 I still hope to improve my scraping capacity…and I really need to figure out to DRY up this process, since it’s a total pain to re-create each of these cells, as I’ve done in the ADB project scraping notebook.

Cheers!!!