Skip to content

Commit

Permalink
Merge pull request #17 from orgrimarr/feature/refactoring
Browse files Browse the repository at this point in the history 
Feature/refactoring
  • Loading branch information
@orgrimarr
orgrimarr authored Nov 14, 2020
2 parents dc0435d + 0933753 commit ff3d547
Show file tree
Hide file tree
Showing 8 changed files with 795 additions and 1,029 deletions.
78 changes: 50 additions & 28 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@
[![Known Vulnerabilities](https://snyk.io/test/github/orgrimarr/node-wetransfert/badge.svg)](https://snyk.io/test/github/orgrimarr/node-wetransfert)

# Changelog
- 2.2.0
- Remove deprecated request-* libs and use node-fetch instead
- Fix wetransfer upload (send emails)
- 2.1.5
- Fix upload (get link)
- Fix download
Expand All @@ -22,7 +25,7 @@ npm install wetransfert --save
or
yarn add wetransfert
```
Tested in node 8.x / 12.x
Tested in node 12.x


## You can require the module like this
Expand Down Expand Up @@ -75,18 +78,22 @@ download(myUrl, myDestinationFolder, ['aaaaaaaaa'])
});
```

# Download weTransfer content from url + pipe response
> /!\ If your transfer contain only one file, wetransfer does not zip the content. Be carefull when using the downloadPipe function. You can obtain all files information using the getInfo function.
# Download weTransfer content from url + pipe response (progress with callback)
### downloadPipe(url)

This function take a valid wetransfer url
This function take a valid wetransfer url. Like the classique download function, you can specify the file ids you want to download. downloadPipe(response.shortened_url, ["fileID"])

It return a Promise and resolve a ReadableStream you can pipe. This stream come from [request-progress](https://www.npmjs.com/package/request-progress). So you can listen for progress while piping
It return a Promise and resolve a ReadableStream you can pipe.

If you need a progress, you can obtain the total size with the getInfo function

## Exemple
``` javascript
const { downloadPipe } = require('wetransfert');

downloadPipe(response.shortened_url)
downloadPipe(response.shortened_url, null)
.then(files => {
files.pipe(fs.createWriteStream("/home/orgrimarr/wetransfer/myDownload.zip"))
})
Expand Down Expand Up @@ -118,35 +125,44 @@ getInfo('myWeTransfertURL')
``` json
{
"content": {
"id": "myID",
"security_hash": "9cc5646",
"id": "cff0151af18a003424fad90a47375f3620201113204655",
"state": "downloadable",
"transfer_type": 1,
"shortened_url": "myShortURI",
"title": null,
"description": "",
"transfer_type": 4,
"shortened_url": "https://we.tl/t-BUr6nd2DAP",
"expires_at": "2020-11-20T20:47:07Z",
"password_protected": false,
"uploaded_at": "2020-11-13T20:47:07Z",
"expiry_in_seconds": 596443,
"size": 497659,
"deleted_at": null,
"recipient_id": null,
"display_name": "flower-3876195_960_720.jpg",
"security_hash": "828b5e",
"description": "Hi this is an upload from https://github.com/orgrimarr/node-wetransfert API",
"items": [
{
"id": "myItemID",
"content_identifier": "file",
"name": "MyFIleName",
"size": 30779833462,
"previewable": false
"id": "579ed7dce3ea1b93a8dff0ee67c0b0e620201113204655",
"name": "flower-3876195_960_720.jpg",
"retries": 0,
"size": 147377,
"item_type": "file",
"previewable": true,
"content_identifier": "file"
},
{

"id": "4d121cf7fb261b2fb2e728afa6a36b7520201113204655",
"name": "gnu.txt",
"retries": 0,
"size": 34667,
"item_type": "file",
"previewable": false,
"content_identifier": "file"
}
],
"password_protected": false,
"per_file_download_available": true,
"expires_at": "2017-09-09T10:22:05Z",
"uploaded_at": "2017-09-02T10:22:20Z",
"deleted_at": null,
"size": 31067650,
"expiry_in_days": 7,
"expiry_in_seconds": 597661
"sessionCookie": "_wt_session=UkJPUmNjZW5EeEpWejlya; domain=wetransfer.com; path=/; secure; HttpOnly; SameSite=Lax",
"csrf": "+dM4tvhVEguYfovUU60pnkK01uaabujp1oAsm8iNe2sf4ZBDeke2cTRR6VNBPZeegSF4fzgKylX+zyeZQEtFeA=="
},
"downloadURI": "myDownloadURI"
"downloadURI": "https://download.wetransfer.com//eu2/cff0151af18a003424fad..........."
}
```

Expand All @@ -161,14 +177,18 @@ If not, it return false
# Upload
You can upload a total file size >= 2Gibibyte (2147483648 Byte)

upload('mailSender', ['receiverMail'], ['file1'], 'myMessage', 'ui_language')
upload('mailSender', ['receiverMail'], ['file1'], 'myMessage', 'ui_language', username, password)

**/!\ Wetransfer upload (send email) is no longer possible without a wetransfer account. Wetransfer add a captcha so i can't script the upload. You can specify yout wetransfer username/password to the upload function**

The upload function parameters :
- mailSender: A valid mail address of the sender
- receiverMail: An array of valid destination address
- file1: An array of valid file path you wan to transfer
- myMessage: The message you want to send
- ui_language: The language of the wetransfer receiver. ex: en, fr
- username: Your wetransfer account username. /!\ username and mailSender email must be the same
- password: Your wetransfer account password

The upload function expose an event emitter and will trigger 3 event :
- progress: Represent the state of the upload
Expand All @@ -177,7 +197,7 @@ The upload function expose an event emitter and will trigger 3 event :

## Exemple
``` javascript
const myUpload = upload('mailSender@gmail.com', ['receive1@gmail.com', 'receive2@gmail.com'], ['D:/Video/MEDIA150212142309947screen.mp4', 'C:/Users/pc/Desktop/toto2.txt', 'C:/Users/pc/Desktop/tata.txt'], 'Hello World', 'en')
const myUpload = upload('mailSender@gmail.com', ['receive1@gmail.com', 'receive2@gmail.com'], ['D:/Video/MEDIA150212142309947screen.mp4', 'C:/Users/pc/Desktop/toto2.txt', 'C:/Users/pc/Desktop/tata.txt'], 'Hello World', 'en', 'username', 'password')
.on('progress', (progress) => console.log('PROGRESS', progress))
.on('end', (end) => console.log('END', end))
.on('error', (error) => console.error('ERROR', error));
Expand Down Expand Up @@ -304,6 +324,8 @@ The upload function expose an event emitter and will trigger 3 event :
If mailSender and receiverMail is equal '', you can upload files without send email.
Remember do not forget get URL in "end" object.

With this mode you dont need a wetransfer account


[End Object](#response-exemple)

Expand Down
140 changes: 65 additions & 75 deletions handler/download.js
View file
Original file line number Diff line number Diff line change
@@ -1,71 +1,58 @@
const { getInfo } = require('./getInfo');
const PProgress = require('../utils/PProgress');
const ReqProgress = require('request-progress');
const request = require('request');
const debug = require('debug')("wetransfert:download")
const mkdirp = require('mkdirp');
const unzip = require('unzipper');
const path = require('path');
const fs = require('fs');
const { getInfo } = require('./getInfo')
const PProgress = require('../utils/PProgress')
const fetch = require('node-fetch')
const debug = require('debug')("wetransfert:download")
const mkdirp = require('mkdirp')
const unzip = require('unzipper')
const path = require('path')
const fs = require('fs')
const stream = require('stream')
const util = require('util')

const streamPipeline = util.promisify(stream.pipeline)


exports.download = function(url = '', destPath = null, fileIds = null){
exports.download = function (url = '', destPath = null, fileIds = null) {
return new PProgress(async (resolve, reject, progress) => {
if(!destPath){
return reject(new Error('Not destination path found'));
if (!destPath) {
return reject(new Error('Not destination path found'))
}
try{
const weTransfertObject = await getInfo(url, fileIds);
if(!weTransfertObject) return reject(new Error('Not a valid url'));
const downloadProcess = ReqProgress(request(weTransfertObject.downloadURI), {
throttle: 500, // Throttle the progress event to 2000ms, defaults to 1000ms
delay: 0, // Only start to emit after 1000ms delay, defaults to 0ms
// lengthHeader: 'x-transfer-length' // Length header to use, defaults to content-length
})
.on('progress', (state) => {
// The state is an object that looks like this:
// {
// percent: 0.5, // Overall percent (between 0 to 1)
// speed: 554732, // The download speed in bytes/sec
// size: {
// total: 90044871, // The total payload size in bytes
// transferred: 27610959 // The transferred payload size in bytes
// },
// time: {
// elapsed: 36.235, // The total elapsed seconds since the start (3 decimals)
// remaining: 81.403 // The remaining seconds to finish (3 decimals)
// }
// }
progress(state.percent.toFixed(2));
})
.on('error', (err) => {
return reject(err);
})
.on('end', () => {
return resolve(weTransfertObject);
})
if((weTransfertObject.content.items.length >= 2) && !fileIds){
if(!fs.existsSync(destPath)){
await mkdirp(destPath)
}
downloadProcess
.pipe(unzip.Extract({ path: destPath }))
try {
const weTransfertObject = await getInfo(url, fileIds)
if (!weTransfertObject) {
return reject(new Error('Not a valid url'))
}

if (!fs.existsSync(destPath)) {
await mkdirp(destPath)
}
else{
if(!fs.existsSync(destPath)){
await mkdirp(destPath)
}
downloadProcess
.pipe(fs.createWriteStream(
path.join(destPath, weTransfertObject.content.items[0].name)
));

const destinationStream = weTransfertObject.content.items.length >= 2 || (Array.isArray(fileIds) && fileIds.length >= 2)
? unzip.Extract({ path: destPath })
: fs.createWriteStream(path.join(destPath, weTransfertObject.content.items[0].name))

const response = await fetch(weTransfertObject.downloadURI)
if (!response.ok) {
throw new Error(`Unexpected response ${response.status} ${response.statusText}`)
}
debug('get total size', parseInt(response.headers.get('content-length')), weTransfertObject.content.size)
const size = parseInt(response.headers.get('content-length')) || weTransfertObject.content.size

let uploadedByte = 0
const progressStream = new stream.PassThrough()
progressStream.on('data', chunk => {
uploadedByte += chunk.length
const percent = uploadedByte / size
progress(percent.toFixed(2))
})

await streamPipeline(response.body, progressStream, destinationStream)

return resolve(weTransfertObject)
}
catch(e){
return reject(e);
catch (e) {
return reject(e)
}
});
})
}


Expand All @@ -74,21 +61,24 @@ exports.download = function(url = '', destPath = null, fileIds = null){
.then(files.pipe(WritableStream))
.catch(console.error)
*/
exports.downloadPipe = async function(url = ''){
try{
debug("downloadPipe", url)
const weTransfertObject = await getInfo(url);
debug("weTransfertObject", weTransfertObject)
debug("weTransfertObject.downloadURI", weTransfertObject.downloadURI)
if(!weTransfertObject) {
throw new Error('Not a valid url');
}
return ReqProgress(request(weTransfertObject.downloadURI), {
throttle: 500,
delay: 0,
})
exports.downloadPipe = async function (url = '', fileIds = null, progressCallback = null) {
if(typeof progressCallback === "function"){
progressCallback = function(){}
}
catch(error){
throw error

debug("downloadPipe", url)
const weTransfertObject = await getInfo(url, fileIds = null)
debug("weTransfertObject", weTransfertObject)
debug("weTransfertObject.downloadURI", weTransfertObject.downloadURI)
if (!weTransfertObject) {
throw new Error('Not a valid url')
}

const size = weTransfertObject.content.size
const response = await fetch(weTransfertObject.downloadURI)
if (!response.ok) {
throw new Error(`Unexpected response ${response.status} ${response.statusText}`)
}

return response.body
}
Loading

0 comments on commit ff3d547

Please sign in to comment.