Posted by In this article, we will update multiple records. If you just put all the records in one go then there is a possibility your program may turn into a getaway timeout error. To overcome this problem we will implement a strategy which will update all entries without causing any error on the browser.
Table of Contents
Need for mass update
Sometimes, if you are uploading several products to some 3rd party website or a marketplace like Amazon, the API takes some time when you send a request and wait for a response. This long duration may result in updating or pushing to the marketplace of around 10 to 20 products. The remaining products won’t do as it will cause timeout error 504.
What is getaway timeout error 504
The error codes which start from 5xx are server-level errors. When a client makes a good request to the server but the server doesn’t complete the job, which means something is wrong with the server. In this article, we are looking into the 504 getaway timeout error. A 504 error means the server did not receive a response fast enough from the server it was making a request to which resulted in timeout. Let’s say when you are updating around 1000 values, the server can’t do all of them in one go.
Solving timeout error
In this article, we are speaking about the mass update of objects. So according to this, we will solve this problem using Batch Processing.
Batch Processing
Batch processing is where a large amount of data is broken down into smaller and equal parts. These parts are the batch which are executed one after other. In the article, we will use products to be uploaded in batches of 5. The focus here is to show batch processing of large numbers of objects the API part of will be just a dummy code.
We will begin these by creating a custom module. For that we need two files. Follow our article which will help to create these two files Create Magento 2 Custom Module Development.
Add mass action on product list
We will create a mass action which will then upload our products. We will need two controllers, one will open a batch processing page and other will upload products this will be done via Ajax. We also need the layout file and the phtml file.
Mass action in Product Listing UI component file product_listing.xml file will be under LearningMagento\Massaction\view\adminhtml\ui_component
<?xml version="1.0" encoding="UTF-8"?>
<listing xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="urn:magento:module:Magento_Ui:etc/ui_configuration.xsd">
<listingToolbar name="listing_top">
<massaction name="listing_massaction">
<argument name="data" xsi:type="array">
<item name="config" xsi:type="array">
<item name="component" xsi:type="string">Magento_Ui/js/grid/tree-massactions</item>
</item>
</argument>
<action name="update_product">
<argument name="data" xsi:type="array">
<item name="config" xsi:type="array">
<item name="type" xsi:type="string">massoperation</item>
<item name="label" xsi:type="string" translate="true">Update product on Marketplace</item>
<item name="url" xsi:type="url" path="learnmagento/product/totalselected"/>
<item name="confirm" xsi:type="array">
<item name="title" xsi:type="string" translate="true">Upload products</item>
<item name="message" xsi:type="string" translate="true">Are you sure want to Upload Products to marketplace?
</item>
</item>
</item>
</argument>
</action>
</massaction>
<paging name="listing_paging"/>
</listingToolbar>
</listing>Controller 1. Totalselected.php This controller will be called from the mass action but this controller won’t perform the actual mass action but it will help to solve the problem of timeout by using the chunk size to control the request not to lead to 504. Create the controller under LearningMagento\Massaction\Controller\Adminhtml\Product. In the below code we are using the Filter class to get the number of products being selected when the action was perform on product page. If we are not getting any products then we are redirecting back to the product listing page with an error.
<?php
namespace LearningMagento\Massaction\Controller\Adminhtml\Product;
use Magento\Backend\App\Action\Context;
use Magento\Framework\App\ResponseInterface;
use Magento\Framework\Exception\LocalizedException;
class TotalSelected extends \Magento\Backend\App\Action
{
const CHUNK_SIZE = 3;
protected $filter;
protected $collectionFactory;
protected $resultPageFactory;
protected $session;
public function __construct(
\Magento\Ui\Component\MassAction\Filter $filter,
\Magento\Catalog\Model\ResourceModel\Product\CollectionFactory $collectionFactory,
\Magento\Framework\View\Result\PageFactory $resultPageFactory,
Context $context
)
{
$this->filter = $filter;
$this->collectionFactory = $collectionFactory;
$this->resultPageFactory = $resultPageFactory;
$this->session = $context->getSession();
parent::__construct($context);
}
/**
* Execute action based on request and return result
*
* Note: Request will be added as operation argument in future
*
* @return \Magento\Framework\Controller\ResultInterface|ResponseInterface
* @throws \Magento\Framework\Exception\NotFoundException
*/ public function execute()
{
try {
$collection = $this->filter->getCollection($this->collectionFactory->create());
$productIds = $collection->getAllIds();
if (count($productIds) == 0) {
throw new LocalizedException(__('No Product selected to process.'));
}
$productIds = array_chunk($productIds, self::CHUNK_SIZE);
$params = $this->getRequest()->getParams();
$params["productids"] = $productIds;
$this->getRequest()->setParams($params);
$this->session->setMarketplaceProducts($productIds);
$resultPage = $this->resultPageFactory->create();
$resultPage->setActiveMenu('LearningMagento_Massaction::marketplace');
$resultPage->getConfig()->getTitle()->prepend(__($params["type"].' Products'));
return $resultPage;
}
catch (\Exception $exception) {
$this->messageManager->addErrorMessage($exception->getMessage());
$this->messageManager->addErrorMessage("Something went wrong while processing the product(s)");
}
return $this->_redirect('*/*/index');
}
}Layout file learnmagento_product_totalselected.xml this file has a block class which we will be using to get the url for the ajax controller class and it has a phtml file for rending.
<?xml version="1.0"?>
<page xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" layout="admin-1column" xsi:noNamespaceSchemaLocation="urn:magento:framework:View/Layout/etc/page_configuration.xsd">
<body>
<referenceContainer name="content">
<block class="LearningMagento\Massaction\Block\Adminhtml\Product\Totalselect" name="learnmagento_product_activity" template="product/process.phtml"/>
</referenceContainer>
</body>
</page>Totalselect block class file should be located under LearningMagento\Massaction\Block\Adminhtml\Product\Totalselect
<?php
namespace LearningMagento\Massaction\Block\Adminhtml\Product;
class Totalselect extends \Magento\Backend\Block\Widget\Container
{
protected $ids;
protected $registry;
public function __construct(
\Magento\Backend\Block\Widget\Context $context,
\Magento\Framework\Registry $registry,
$data = []
) {
parent::__construct($context, $data);
$this->_getAddButtonOptions();
$this->registry = $registry;
$this->ids = $this->registry->registry('productids');
}
/**
* Back button
*/ public function _getAddButtonOptions()
{
$splitButtonOptions = [
'label' => __('Back'),
'class' => 'action-secondary',
'onclick' => "setLocation('" . $this->_getCreateUrl() . "')"
];
$this->buttonList->add('add', $splitButtonOptions);
}
/**
* Ajax call for all type of product API calls
*
* @return false|string
*/ public function getAjaxUrl()
{
return $this->getUrl('learnmagento/product/upload');
}
/**
* Get selected product ids
* @return array|mixed
*/ public function getSelectedProducts()
{
$params = $this->getRequest()->getParams();
if(isset($params["productids"])) {
return $params["productids"];
}
return [];
}
}process.phtml file. In this file I am using the two js libaries LineProgressbar and accordion. In the below code it will call the ajax controller every time one request is finished till all the products are being processed. This way there won’t be any gateway timeout error nor 504 error.
<?php
$productIds = $block->getSelectedProducts();
$total = count($productIds);
?>
<div class="row">
<div class="col-md-12" style="margin-top: 10px;">
<div class="panel panel-default">
<div class="block-content panel-body ">
<div id="learnmagento-progress-bar"></div>
<br>
<div id="batches">
<div class="batches-tab" data-role="collapsible">
<div data-role="trigger">
<span>Responses</span>
</div>
</div>
<div class="batches-content" data-role="content">
<ul id="profileRows" style="list-style: none;">
<li style="list-style: none;">
<?php echo 'Total ' . $total . ' Batch(s) Found.'; ?>
</li>
<li style="list-style: none;" id="updateRow">
<span id="updateStatus" class="text"><?php echo __("Updating..."); ?></span>
</li>
<li id="liFinished" style="display:none; list-style: none;">
<?php echo __("Finished product Processing."); ?>
</li>
</ul>
</div>
</div>
</div>
</div>
</div>
</div>
<style>
#batches {
border: 1px solid #ddd;
border-radius: 0;
}
.batches-tab {
background: #eee;
padding: 1rem;
cursor: pointer;
font-weight: bold;
&
:first-child {
border-bottom: 1px solid #ddd;
}
&
:nth-last-child(2) {
border-top: 1px solid #ddd;
}
}
.batches-content {
padding: 0.5rem 0.5rem;
}
/*li*/ #batches ul {
list-style: none;
padding: 0;
margin: 0;
}
#batches ul li {
vertical-align: middle;
padding: 2px 2px 2px 2px;
font: normal 12px sans-serif;
font-size: small;
}
#batches li img {
vertical-align: middle;
margin-right: 5px;
max-width: 12px;
}
#batches span {
vertical-align: middle;
}
</style>
<script>
require([
'jquery',
'jquery/ui',
'accordion',
'lineProgressbar'
],
function ($) {
$("#learnmagento-progress-bar").LineProgressbar({
percentage: 0,
fillBackgroundColor: '#77a21b',
height: '25px'
});
$( ".batch-container" ).accordion({ collapsible: true, active: false});
var totalRecords = parseInt("<?php echo (int)$total; ?>");
var countOfSuccess = 0;
var id = 0;
var liFinished = document.getElementById('liFinished');
var updateStatus = document.getElementById('updateStatus');
var updateRow = document.getElementById('updateRow');
var newArr =<?php echo json_encode($productIds); ?>;
//call on load
sendRequest();
function sendRequest() {
//update progress
$("#learnmagento-progress-bar").LineProgressbar({
percentage: parseInt(((id + 0.5) / totalRecords) * 100),
fillBackgroundColor: '#77a21b',
height: '35px',
duration: 0
});
updateStatus.innerHTML = (id + 1) + ' Of ' + totalRecords + ' Processing';
var request = $.ajax({
type: "GET",
url: "<?php echo $block->getAjaxUrl();?>",
data: {productids: newArr[id]},
success: function (data) {
var json = data;
id++;
var span = document.createElement('li');
if (json.hasOwnProperty('success')) {
// For api related calls
if(json.success.hasOwnProperty('response') && json.success.hasOwnProperty('status_code')) {
if(json.success.response !== null && json.success.status_code === 200) {
span.innerHTML =
'<span>' + json.success + '</span>';
span.id = 'id-' + id;
updateRow.parentNode.insertBefore(span, updateRow);
}
else {
span.innerHTML =
'<span>Error ' + json.success.status_code + '</span>';
span.id = 'id-' + id;
updateRow.parentNode.insertBefore(span, updateRow);
}
}
else {
span.innerHTML =
'<span>' + json.success + '</span>';
span.id = 'id-' + id;
updateRow.parentNode.insertBefore(span, updateRow);
}
countOfSuccess++;
}
else {
if (json.hasOwnProperty('error')) {
var heading = '<span>'+json.error+'</span>';
var errorTemplate = '<div class="batch-container">' +
'<div data-role="collapsible" style="cursor: pointer;">' +
'<div data-role="trigger">' + heading + '</div></div>' +
'</div>';
}
span.innerHTML = errorTemplate;
span.id = 'id-' + id;
updateRow.parentNode.insertBefore(span, updateRow);
$( ".batch-container" ).accordion({ collapsible: true, active: false});
}
},
error: function () {
id++;
var span = document.createElement('li');
span.innerHTML = '<span>Something went wrong </span>';
span.id = 'id-'+id;
updateRow.parentNode.insertBefore(span, updateRow);
},
complete: function () {
//update progress
$("#learnmagento-progress-bar").LineProgressbar({
percentage: parseInt(((id) / totalRecords) * 100),
fillBackgroundColor: '#77a21b',
height: '35px',
duration: 0
});
if (id < totalRecords) {
sendRequest();
} else {
var span = document.createElement('li');
span.innerHTML =
'<span id="updateStatus">' +
totalRecords + ' product batch(s) successfully Procced.' + '</span>';
liFinished.parentNode.insertBefore(span, liFinished);
document.getElementById("liFinished").style.display = "block";
updateStatus.innerHTML = (id) + ' of ' + totalRecords + ' Processed.';
}
},
dataType: "json"
});
}
function parseErrors(errors) {
var data = (errors);
var result = {
'status': true,
'errors': ''
};
if (data) {
result.errors = '<table class="data-grid" style="margin-bottom:10px; margin-top:10px"><tr>' +
'<th style="padding:15px">Sl. No.</th>' +
'<th style="padding:15px">Sku</th>' +
'<th style="padding:15px">Errors</th></tr>';
var products = Object.keys(data).length;
var counter = 0;
$.each(data, function (index, value) {
var messages = '';
$.each(value.errors, function (i, v) {
if (typeof v === 'object' && v !== null && Object.keys(v).length > 0) {
messages += '<ul style="list-style: none;">';
$.each(v, function (attribute, err) {
messages += '<li><b>'+attribute+'</b> : '+err+'</li>';
});
messages += '</ul>';
}
});
if (messages === '') {
counter++;
messages = '<b style="color:forestgreen;">No errors.</b>';
}
if (!value['Field']) {
value['Field'] = value['SellerSku'];
}
//var sku = "<a href='" + value.url + "' target='_blank'>" + value.sku + "</a>";
result.errors += '<tr><td>' + (value['Field']) + '</td><td>' + (value['SellerSku']) + '</td><td>' + (value['Message']) +
'</td></tr>';
});
result.errors += '</table>';
if (products === counter) {
result.status = false;
}
}
return result;
}
function getPercent() {
return Math.ceil(((id + 1) / totalRecords) * 1000) / 10;
}
}
);
</script>
Controller 2. Upload.php this class will be present in LearningMagento\Massaction\Controller\Adminhtml\Product
<?php
namespace LearningMagento\Massaction\Controller\Adminhtml\Product;
use Magento\Framework\Exception\LocalizedException;
class Upload extends \Magento\Backend\App\Action
{
protected $resultJsonFactory;
protected $resultPageFactory;
protected $productHelper;
public function __construct(
\Magento\Backend\App\Action\Context $context,
\Magento\Framework\View\Result\PageFactory $resultPageFactory,
\LearningMagento\Massaction\Helper\Product $product,
\Magento\Framework\Controller\Result\JsonFactory $resultJsonFactory
) {
parent::__construct($context);
$this->productHelper= $product;
$this->resultJsonFactory = $resultJsonFactory;
$this->resultPageFactory = $resultPageFactory;
}
public function execute()
{
$resultJson = $this->resultJsonFactory->create();
try {
$params = $this->getRequest()->getParams();
if (!isset($params["productids"])) {
throw new LocalizedException(__("Product id is missing in the request"));
}
$response = $this->productHelper->uploadProducts($params["productids"]);
if (isset($params["productids"]) && $response) {
return $resultJson->setData(
[
'success' => count($params["productids"]) . " Product(s) Uploaded successfully",
'messages' => $response
]
);
}
return $resultJson->setData(
[
'error' => count($params["productids"]) . " Product(s) Upload Failed",
'messages' => $response
]
);
}
catch (\Exception $exception) {
return $resultJson->setData(
[
'error' => "Product(s) Upload Failed",
'messages' => $exception->getMessage(),
]
);
}
}
}The above class will be used to upload products to a marketplace. I have not added the code for the helper class as you can do it from your side. The aim for this article was to demonstrate a way to prevent gateway timeout error using the mass update with batch processing.